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Introduction 



Title X of the Elementary and Secondary Education Act passed by 
Congress in 1965 was the first educational legislation in history to call 
for an evaluation at the local level to determine the effectiveness of 
programs generated by federal funds. As such, it created tremen- 
dous demands on the recipients of these funds. 

Title I was put into effect almost overnight with the intent of 
compensating for the limited experience of poor, deprived, and handi- 
capped children. Most districts had little or no time to plan their 
ESEA projects, much less to design project evaluations. Fesearch 
staffs and facilities were inadequate, and staff added subsequently did 
not have the advantage of being involved from the start. Basic data 
suddenly required were not available. Much had to be done to define 
the scope of the evaluation task, to invent useful data gathering instru- 
ments, and to secure information on short notice from those who were 
deeply engaged in project operations. Yet, evidence of project activity 
and effectiveness and evaluations capable of providing such evidence 
were needed in order to gain congressional support for the fiscal year 
1967-1968. It was no doubt with a sense of urgency thai Office of 
Education decision makers searched for existent evaluation designs 

and strategies. 

The position taken by the big cities on ESEA evaluation was of 






special importance to the Office of Education. Most of these cities 
operated with a considerable degree of independence from state and 
federal guidelines* The big cities serve a relatively large number of 
underprivileged students, and most possess research staffs presum- 
ably capable of supporting a serious interpretation of the evaluation 
clause of ESEA. Therefore, a knowledge of big -city evaluation activ- 
ity was particularly important to O. E. It was for this reason that the 
Pittsburgh Evaluation Model Validation Project was amended to in- 
clude a conference for discussion among research directors of some 
of the big cities. Accordingly, a conference was held in Fort Lauder- 
dale, Florida, December 27-30, 1966 with selected representatives 
from the big cities, consultants, and O. E. observors in attendance. 

The organizing theme of the conference was the sharing of evalu- 
ation assumptions and operational procedures. All parties to the con- 
ference were surprised at their agreement on certain ^valuation pur- 
poses although great variation in evaluation methodology obtained. On 
the third and final day of the conference all participants agreed as to 
the worth of a position paper which would attempt to summarize and 
explicate those areas of agreement as well as document samples 
of evaluation work in a number of big-city systems in an attempt to 
better understand local evaluation strategy and intent. Evaluation 
work in two of the big -city systems involved the development and use 
of evaluation models. 
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this publication resulted from the decisions described above and 
is, accordingly, divided into three major areas: principles of evalua- 
tion, evaluation based on theoretical models, and evaluation case 



studies 



Part I 



AN OVERVIEW OF ESEA EVALUATION 



It is apparent that Title I evaluation can serve at least three 
major purposes: quality control, cost benefit analysis, and improved 
decision making. Quality control requires the establishment of pro- 
cedures to monitor and modify programs so as to insure uniform 
products which meet acceptable standards. Cost benefit analysis 
examines the relationship between cost of a program and the value 
of its products relative to the cost of other products of similar value. 
Finally, improved decision making is possible when new sources of 
information about either benefits, costs, program quality, or program 

feasibility are at hand. 

The last of these purposes of evaluation appears to be most 
widely recognized to date. Yet. there seems to be general agreement 
in the Office of Education that current information about Title I programs 
is so minimal as to preclude even a description of Title I program 
activity in most American schools. Information as to benefits, costs, 
and program operations is simply not available. The possibility of 
comparing educational programs on a cost effectiveness basis is still 
but a Defense Department dream in the minds of a few Office of 

Education executives. 

The first purpose, the desire to maintain quality control stand- 
ards for Title I project, through evaluation apparatus, appears to be 
uppermost in the mind, of a good many Office of Education staffers 




as well as the cause of continual effort and frustration. Generally, 
Office of Education personnel have sought to insure program quality 
by establishing categorical restrictions on fund use and by urging 
that more sophisticated evaluation designs be incorporated in program 
proposals. 

This latter approach to evaluation assumes that school districts 
will be more likely to attend to program quality if they know that 
program products will be rigorously appraised. This strategy nec- 
essarily emphasizes the definition of program outputs in quantified 
terms suitable for use in experimental designs. In this strategy the 

government has been abetted by '’research types" in universities. 

« 

And herein lies an unrecognized paradox: as state and federal govern- 
ment officers seek to encourage program quality by means of rigorous 
product assessment, they may thwart quality control work at the only 
point it will be effective— the local level. This local work is usually 
of the process assessment type in which evaluators systematically 
collect and weigh data descriptive of ongoing program activity. 

The desirability of submitting both Title I products and processes 
to evaluation so as to guide policy makers at the national level will 
find universal support. However, it is our belief that process evalua- 
tion for program improvement must occur at the local level before 
national assessments of Title 1 can be meaningful. 
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All Title I programs are new programs in the sense that they are 
new to the school district and to most of the personnel involved. These 
programs are in a ’’becoming stage" for staff. Procedures must change 
with experience. And as procedures change, as the possible and im- 
possible are sorted out, goals must change too. However, when goals 
have been fixed, when students have been frozen into experimental and 
control groups to satisfy rigorous evaluation designs which assume 
stable treatments, the dynamics of essential program change are con- 
stricted out of existence. 

Education may be defined as the art of accurately predicting 
changes in human behavior through applied social science. Because 
methodology is uncertain and conditions are obscure, predictions are 
often inaccurate. When dealing with underprivileged students, about, 
whom we know even less than the average school client, our predic- 
tions are notoriously unreliable. It follows that educational programs 
devised for the underprivileged are unreliable. Programs with the 
same name can differ more from class to class than do programs with 
different names. Upon inspection, Miss Brown’s remedial reading 
program may turn out to be a group therapy class. Miss Smith’s 
therapy class may turn out to be a high school psychology lecture 
series. 

Clearly, the task defined by Title I legislation is to increase 










education program payloads for the underprivileged and to prove it. 
Before this is possible, however, we will have to improve our pro- 
gram development procedures and know that we have improved them. 

It is important to note that improvement in procedure is not always 
immediately reflected in performance. For example, in aviation a 
change in wing design may be ineffective until coupled with increased 
horsepower. So in education a change in instructional material may 
be inconsequential until coupled with a new mix of students or a new 
teaching technique. 

We must be able to demonstrate that procedures exist for the 
improvement of programs resulting in educational benefit to the under- 
privileged. Some program development work in universities, regional 
labs, and centers goes forward. However, it is obvious that unless 
massive aid is given to research in program development work, little 
change in practice will be seen in this decade. That federal policy 
reflects this reality can be inf*, “red from the emphasis being given to 
Title III of ESEA. Title HI is the "tinker and try" title of ESEA . How- 
ever, the $75, 000,000 available for Title III in 1966-1967 runs the 
risk of dilution because it is being spread across the entire spec- 
trum of the American public schools population in contrast to the 
over a billion dollar Title I program funds directed toward those 
with greatest need. Furthermore, O.E.’s present policy appears to 
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be to encourage Title III dissemination projects over developmental 
projects* The assumption that something of value to disseminate exists 
may in many cases not be warrantee because so few programs have 
been properly developed. 

Ultimately, programs will improve only if teachers, administra- 
tors, and students in most of America's classrooms become involved 
in a comprehensive effort to review and improve their work. Such an 
effort requires a careful study of present procedures, a detailed 
analysis of program events and their sequence, and the designing of a 
series of small experiments to test the value of program components. 

Shades of Action Research? Maybe. But there is considerable 
evidence in both industry and education that only when personnel re- 
sponsible for conducting a program are involved in its examination 
and revision will the program improve and endure. To improve educa- 
tional programs we must use Title I to give school personnel a sense 
of freedom to admit error, to revise programs, and to creatively risk 
failure secure in the belief that continuous program evaluation will 
eventually provide success. 

In this light, the fact that there has been little opportunity for 
carefully planned Title I programs is not as serious as some have 
claimed. Programs can be planned after they have begun, "in mid- 
stream" program planning has the advantage of prpviding staff with 
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information based on pragmatic consequence rather than speculation. 
That is, classrooms become labs for culling program development in- 
formation. Further, it becomes possible for an entire program staff 
to contribute to a policy decision and thereby feel a* sense of responsi- 
bility for its consequences. These benefits, of course, ensue only 
when "in midstream" program planning is built in to a local Title I 
effort--that is, when evaluation is interpreted to mean staff self- 
evaluation and continuous program assessment. 

The broadening of the superintendent's decision-making base to 
include program staff decisions may be viewed by some as a loss 
of administrative power. On the contrary, one of the most important 
considerations in an administrator's policy-making deliberations is 
the amount of staff support a decision -will command. For this reason, 
superintendents often establish committees or councils as a sounding 
board for policy prior to enactment. One great disadvantage of such 
sounding boards is that they must react to a hypothetical issue without 
benefit of total staff reaction to "the real thing." When a superintendent 
can involve his entire staff in actual program revision and policy- 
making experience, his decisions as the chief school officer are likely 
to be realistic and enduring. 

When Title I evaluation is used as a mechanism for the develop- 
ment and improvement of school programs, it takes on the appearance 
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of a staff training strategy. Staff activity includes systematic study 
of program variables such as student entry behavior, student- teacher 
interaction, pupil performance on interim tasks, and student and 
teacher indices of attitude and satisfaction. Evaluation becomes a 
vehicle for training staff to meet the changing demands of project 
activity. Such training will eventually serve all information needs 
for process and product evaluation. It also provides continuous data 
on staff, pupils, and program for management decision making. It 
reveals new independent variables and stabilizes program variables, 
both of which are essential to eventual program benefit analysis. 
Judgment about program benefit can be based on product assessment 
only when a program is stable— when it continues to be what staff and 
administration think it is. The products of stable programs can be 
compared only when all of the factors which can change the product, 
other than program, have been identified and taken into account. There 
fore, to expect to evaluate a program on the basis of its products be- 
fore evaluating its processes will generally lead to frustration and 

failure,, 

Yet, process and product evaluation need not be incompatible in 
the same evaluation project. Only planning of event sequence and 
patience are needed; Program development as a function of “in mid- 
stream" planning and training must come first. Experimental design 
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work must be deferred* To expect significant shifts in criterion scores 
before the final form of a program has evolved, is to expect a cater- 
pillar to fly. 

When program is defined by staff as a changing dynamic which . 
must undergo developmental stages of growth, it becomes possible to 
understand, measure, and place a value on any program at any time 
without compulsive reference to standardized achievement test scores 
or other quantified criteria* Scores are not to be discounted in either 
the short term or long term, but they must lose their sacredness as 
the criterion of program effectiveness* Only when applied to the last 
stages of program development are they meaningful, and then only when 
they are part of an experimental design which makes provision for 
many complex factors. 

Some officials in the Office of Education are obviously aware of 
the need to de-emphasize short-term product assessment so that in- 
ternal local evaluation processes may be employed to strengthen 
programs. Yet pressure from the Congress and elsewhere may be 
too great to resist* A similar pressure exists locally from political 
action groups which in their desire to see tangible student benefits . 
do themselves a disservice by ignoring the complex, maturational 
nature of any new school program* 



Part IT 



SUMMARY OF EVALUATION PRINCIPLES 







Distinction Between Goals and Objectives 



Good evaluation starts with a clear definition of goals, or long- 
range educational aims. The goals of the educational enterprise are 
the projected results of the individual programs which constitute the 
entire enterprise^ As such, goals are determined by the educational 
institution in the context of community, society, and projected future 
needs. 

Objectives are the intermediate ends which lead to the realization 
of goals. Thus they are of greater concern to the evaluation team than 
are goals since the objectives provide the bases for the development of 
programs as well as methods used to measure program effects. While 
the value of program objectives is not a concern of the evaluation team, 
the evaluation process may be used to clarify and define objectives, 
and thereby increase their relevance and practicality. 

Information Collection 

Generally speaking, information collection includes the time ■ 
scheduling of information requirements, the selection and/or develop- 
ment of appropriate instruments, a determination of sampling methods 
to be used, the administration of instruments under standard con- . 
ditions, and related activities. Flexibility of methods, techniques, and 
research design must be maintained so that they are appropriate for 
each of the programs evaluated and for the particular stage at which • 
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information is collected for each program. 

The collection of information descriptive of an ongoing program is 
an important early step in evaluation. The characteristics of students 
entering a program, their competencies, attitudes, needs, and 
interests, the special activities of students, teachers, and adminis- 
trators participating in the program- -all these must be determined, 
and the information should be quantified if possible. 

Generally, adequate instruments do not exist for the description 
and recording of student, teacher, and administrator behavior. Most 
school districts must either distort phenomena to fit standardized 
commercial instruments or must use ad hoc questionnaires without 
evidence as to their reliability. In either case, a gross distortion of 
fact is likely to result. Even the reporting of simple attendance figures 
may be subject to errors, especially if these data are collected and 
summarized by a person who is more interested in artistry than in 
statistics. Close attention must be given to the creation of forms for 
supplying data and to the directions accompanying these forms. If 
accuracy is desired, the collection and tabulation of statistical data 
should be an established procedure controlled* by the evaluator. 
Summary reports prepared by someone else may not stand actual 
s crutiny. 
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Info rmation Reporting 



Once information has been collected, the evaluator is responsible 
for the compilation of data and the production and editing of evaluation 
reports, Including the necessary tables and graphs. The report 
format may be a joint undertaking of the evaluation and operations unit. 
Complete evaluation reports or appropriate summaries should be 
distributed to all participating distrw.v personnel, district adminis- 
trators, board members, state and federal agencies, and other 
interested groups. The feedback of evaluation information to the 
field staff, particularly to teachers, is of vital importance. 

These diverse groups may require reports written for different 
purposes. The Superintendent and Board may have interests in an 
evaluation quite different from those of a program staff. Public 
special interest groups may be concerned only with the success of 
special groups of children, and they may narrowly define success as 
dramatic changes in performance on standardized tests. Students 
themselves may apply very different but reasonable criteria to the 
evaluation program: their own interests, enthusiasm, and attitude 
toward school. The parents of these children may have similar evalu- 
ation interests. 

Regardless of the purpose of the report, the timing of dissemi- 
nation is crucial. Evaluation reports, perfect though they may be, 
submitted one year or even one day after a program has been refunded 



are of little value to teachers and administrators and of little interest 



to anyone else. A perceptive but unsophisticated report that does not 
follow the canons of academic research may be subject to scholarly 
criticism, but is of more value when submitted in time to effect 
changes in an ongoing program than is a technically perfect report 
submitted too late. Quality in written reports is highly desirable, but 
the report must also be made available in time to influence persons 
party to or interested in the public schools. 

Elements of Support Necessary to Evaluation . 

Climate 

Probably more important than any other single factor to the suc- 
cess of evaluation is that there be a climate which encourages investi- 
gation throughout the school system. Many evaluation efforts fail, or 
succeed only minimally, because teachers and administrators lack 

understanding of, and therefore commitment to, the role of research 

♦ 

in education. Instruction in the rationale for evaluation, research 
procedures, data processing, and data usage must be provided in such 
a way as to truly involve all sectors of the school system in the evalu- 
ative effort. Such in-service training would help to keep alive a 

conscious awareness of common purpose; such awareness, unfortu- 

♦ 

nat'jly, often becomes submerged in the day-to-day operations of 
diverse programs. 
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A climate which encourages inquiry can make evaluation a continu- 
ing activity and an integral part of the educational process. Teachers 

should be encouraged to raise critical questions and then search for 

» « 

answers in systematic ways* The more active teachers are in evalu- 
ation, the more likely pupils are to evaluate their own work. Only if 
the spirit of inquiry is established and maintained will teachers, ad- 
ministrators, and pupils be convinced of the value of filling out another 
time-consuming questionnaire. Certainly, the validity of evaluative 
data is open to serious question if they have to be gathered by forced 
intervention. 

Administrative Philosophy and Structure 

Basic to a good climate for thoughtful inquiry in education is an 
administrative philosophy that is committed to evaluation. Today con- 
tinuing evaluation of established programs is needed, as well as evalua- 
tion of projects that are experimental or innovative in nature. Procedures, 
methods, facilities, and learning materials need to be regarded as means 
to an end, and subject to continual improvement. It may well be that 
existing programs that were good in other times or under different 
conditions no longer yield the same benefits for pupils they once did. 

More enlightened doubt about present practices will lead to more con- 
victions and more innovations. 

Also embodied in this philosophy is the disposition to establish 
two-way communication with concerned public and professional personnel; 
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to recognize and use findings of previously completed research of 
other cities and other agencies; and to cooperate in common research 
projects of other large cities* 

There are a variety of school district administrative arrangements 
that lend themselves to the assumption of the responsibilities attendant 
to the development, operation, and evaluation of educational programs* 

A school district may use its existing administrative organization, it 
may establish a separate evaluation unit, or it may contract with an 
outside agency* 

Whatever the structure decided upon, the research director should 
be organizationally placed so that he is able to interact freely and fre- 
quently with associate superintendents or ether line and staff personnel 
who are able to mobilize the resources of the school district both within 
and across organizational substructures* He should also have immediate 
access to the general superintendent of schools or persons equally em- 
powered to render final judgments* Moreover, these relationships should 
be characterized by a high degree of mutual professional trust; 

The evaluation unit itself should not be subordinate to those in charge 
of the program being evaluated* Kvaluation personnel should serve in a 
staff or advisory capacity to project personnel rather than reporting 
"in line” to them* Such organizational placement is essential in order 
for independent, valid evaluations to be obtained* 
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Budget 

Administrative philosophy and structure play an important role 
in insuring that adequate funds are supplied for evaluation activity, 
although just what adequate budget is cannot be prescribed. Some 
cities are presently using about 3 percent of total ESEA Title I funds 

for evaluation purposes. Certainly 5 percent of such funds could be 

♦ 

considered reasonable. In fact, evaluation on a broad scale, applying 
to regular school programs as well as to innovative ones, would be 
greatly enhanced if that percentage of the school system’s total opera- 
tional budget were scheduled to research--including the associated 
aspects of evaluation, development, and demonstration. 

Regardless of the amount, an important budget consideration in 
research is internal flexibility. There need to be opportunities to make, 
changes within a total allocation to provide unds for special contin- 
gencies --contract services, in-service training, new learning materials, 
unusual equipment, and developmental work for the exploration of new 
ideas. 

Personnel 

Adequate budget, in part, provides for adequate personnel. A 

crucial need exists for well -trained research and evaluation personnel 

* • 

who are senstive to the sociological, psychological, political, and 
economic demands of our society so that they may be of maximum help 
in context assessment; who are technically competent in research, 
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statistics, and measurement to assist with input assessment; who are 

administratively skilled in working inter personally with a wide variety 

♦ 

of general and technical personnel to assist in process assessment; 
and who are professionally sensitive to the problems of the schools 
in our present society to assist with product evaluation and policy 
making. 

In view of this great need, those persons already within a school 
system, who know it well from practical experience, need to be en- 
couraged to develop their research interests through further training 
with the knowledge that there will be opportunities for leadership in 
research and evaluation in their school systems. Present research 
staff should be up-graded through internship programs. Persons 
trained in academic research who are brought into the schools must 
be trained again- -in the practical use of their theory under noncontrolled 
conditions. Formal course work in methodology of research and sta- 
tistical techniques is a necessary aid to evaluation, but university 
credits and degrees do not make a research worker or an evaluator 
unless they are combined with the ability to apply these skills in a 
public school setting. 

Facilities 

Not only does adequate budget provide for personnel, it must also 
include provisions for facilities. With an operation as extensive and 
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complex as evaluation has become, many things are required: offices, 
office machines of all types, warehouses and storage space-, conference 
rooms, and space for experimental utilization of equipment materials 
and methods. Since these kinds of facilities are usually available in 
one form or another, attention must now be turned to a more crucial 
present need --improved school system facilities and capacities for in- 
formation management. 

Retrieval of information for analysis is becoming more difficult 
as the volume of relevant data increases. Special data processing 
systems should be planned and made operational for exclusive use in 
managing information related to instruction. Such systems must be 
sophisticated enough to employ efficient methods for data collection and 
use. Random access capabilities are required so as to be able to draw 
from vast data banks of pupil, teacher, and school system information. 

Effective evaluation and communication are dependent upon a com- 
petent technology. Methods of data retrieval and storage currently in 
use are hopelessly antiquated. For example, much information is 
presently available on all students, including testing results, teacher 
evaluations, counsebr records, extracurricular interests, and family 
histories. Yet because such data are recorded on so many forms and 

stored in so many different places, they are difficult, if not impossible 
to retrieve. Computer technology is absolutely essential for the utilization 

of all information sources. Only through an adequate data processing 
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system can all potential and existing information be collected, synthesized, 
made immediately accessible, and utilized to fullest advantage. 

New Directions 

A desirable outcome of required federal program evaluation in large 

school systems is the movement toward the formal evaluation of all in- 

\ 

structional programs. Through experience with Title I evaluations, it 
has become apparent that the usefulness of these evaluations is limited 
due to the lack of comparable information about all other instructional 
programs. This problem is especially acute in the larger school systems 
where it is impossible to partial out the effects of one project on a person 
who is being subjected to multiple influences from a variety of projects. 
Furthermore, interventions such as ESEA ultimately affect all other 
elements of the school system. These overall effects cannot be assumed 
to be uniformly positive. Therefore, there is a demonstrated need for 
continuous system-wide evaluation of all educational programs by a single 
research and evaluation office. 

There is need for caution on the part of those who believe that re- 
search units will suddenly realize their potential for research and evalua- 
tion, Time, trained minds, and adequate supporting staff and facilities 
are all needed if significant and helpful service is to be given. It probably 
is worth repeating that in the process of solving evaluation problems, 

more problems will be discovered. Ultimately, however, each school 
district in America should be able to boast the benefits of a well defined 
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and unified researched evaluation function. -be it performed by one man 
or a large staff. Title I of the 1965 ESEA may be forgotten as a poverty 
act but long remembered as the source of systematic self-appraisal in 



America’s schools* 










Chapter 2 

EVALUATION BASED ON THEORETICAL MODELS 















Parti 



THE COLUMBUS EVALUATION PROGRAM 







I, THE EVALUATION CENTER AND ITS PRINCIPLES OF OPERATION 

Introduction; A Collaborative Relationship in Evaluation 
The evaluation program currently being conducted in Columbus, 
Ohio, is a collaborative effort between the Ohio State University s 
Evaluation Center and the Columbus Public Schools. The collabo- 
ration is based on joint support by the University and Columbus Public 
Schools, with each organization contributing both funds and personnel 
to the effort. The evaluation program is focused on Columbus' Title I 
program established under the Elementary and Secondary Education 
Act of 1965. 

As might be expected in such a collaboration between university 
and public agencies, the three-year project has several general 
purposes, designed to meet the particular needs of both organizations 

involved. Among these purposes are: 

first , to provide to the Columbus Schools information to design, 
operate, and assess the impact of the Title I program and to make 
appropriate evaluation reports to the State Department of Educa- 
tion and the U.S. Office of Education; 

second, to give to the Columbus Schools a core of persons experi- 
enced and trained in evaluation, so that the school system can 
staff an evaluation agency within the system; 
third, to provide the Evaluation Center with an opportunity. to 
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develop and test techniques, instrument.;, and designs for 
evaluation; 

and fourth, to provide the Evaluation Center with a laboratory to 
practice in the public schools. 

The general strategy for the evaluation program in Columbus has 
two major components. First, it includes the conduct of research and 
development in evaluation. Second, by using staff from OSUEC and 
personnel on leave from the Columbus Public Schools, it includes 
simultaneously provision of evaluation information for an ongoing 
Title I program in a school environment. Persons from OSUEC and 
Columbus Public School personnel gather data on the context within 
which projects operate, on the operation of projects from day to day, 
and on the impact of projects on the behavior of inner-city students. 

At the same time, the staff members are involved in research and 
development in evaluation and are being trained to take a position of 
leadership in evaluation upon joining or rejoining the school system. 

The Role of the Evaluation Center in the Columbus Project 
The Evaluation Center is an agency of the Ohio State University's 
School of Educ?tion dedicated to advancing the science of evaluation. 

As conceived by the Center, evaluation is the process of providing 
information which is essential to decisions made in planning, program* 

i * 

ming, and implementing activities to meet educational goals. The 

- 32 - 



ERjt 






Columbus 
Page 3 



Center has the following as its objectives: (1) to study planning and 
evaluation in education; (2) to develop models and methods for evalu- 
ation project designs, project activities, and project outcomes; 

(3) to develop methods and materials for implementing evaluation 
programs; (4) to diffuse information related to evaluation; (5) to help 
practitioners effectively use evaluation designs and tools; and (6) to 
provide instruction in evaluation. These objectives are reflected in 
the major activities of the Center which include research, development, 
training, diffusion; and limited service in evaluation. 

A Conceptual Model of Evaluation 
The evaluation efforts undertaken in the Columbus project were 
guided by a model formulated by Dr. Daniel Stufflebeam, Director of 
the Evaluation Center. This model is based on the concept_thatthe 

r evaluation is to provid e information for making decisio ns. 

Pupil attainments are measured for detecting problems and specifying 
objectives in proposals for financial assistance; funding agencies 
obtain the judgments of experts about the quality of proposals before 
deciding to fund or reject them; project managers obtain progress 
reports before deciding to modify plans or procedures in process; 
legislative bodies require data about the effectiveness of programs 
before deciding to continue, discontinue, or change the level of funding 

for such programs. These examples illustrate that key decision 

• * 
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makers in a change such as the Columbus program process continually 
need valid evaluative information. To serve such decision making 
requirements effectively, evaluation must be valid (homomorphic to 
the variables of interest), reliable (reproducible), timely (available 
when the decision maker needs it), credible (trusted by the decision 
maker and those he must serve), and accessible (available to those who 
have a need to know). 

Since there are many kinds of decisions associated with and re- 
lated to change activities, there are also different types of evaluation. 
The nature of decisions involved in planning and implementing change 
activities encompasses four general types of decisions and suggests 
that the needed evaluation may be conveniently divided into four 
generalized stages. These stage s - - context, input, process, and 
product --comprise the CIPP EVALUATION MODEL, which is depicted 
in Figure 1. 

Context Evaluation 

The first stage of the CIPP Model is context evaluation . Its 
major objective is to locate, delineate, and assess the importance of 
problems and needs in the environment where change is to occur. 

This is accomplished through determining, measuring, and analyzing 
the total system within which change is to occur. It focuses upon the 
setting and ranking of objectives for a system, the intended outputs, 
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differences between intended and actual outputs, and relationships 
between intended and actual outputs, and relationships between actual 
inputs and outputs. It thus pinpoints needs and further suggests 
problems of system design or operation which underlie a state of need. 

In the change process, context evaluation provides information for 
deciding upon the appropriate point of entry to the change process. 
Decisions based upon context evaluation usually result in problem 
statements contained in proposals in certain critical areas. 

Input Evaluation 

To determine how to cope with the identified needs and problems, 
it is necessary to conduct an input evaluation. The objective of input 
evaluation is to identify pos sible solution strategies or coping mecha- 
nisms and to forecast their strengths and weaknesses for solving the 
specified problem within the relevant operating context. Input evalu- 
ation is guided by the point of entry to the change process suggested 

• * 

by context evaluation- -whether research, development, diffusion, or 
adoption activities are most appropriate. The method of input evalu- 
ation is to determine the relevance of alternative courses of action to 
the problem of interest, their potential procedural barriers, the 
possibility of overcoming and the consequences of not overcoming the 
procedural impediments, and the benefits and costs of overcoming 
them. In the change process, input evaluation provides information for 
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deciding upon program strategies and tactics. Decisions based upon 
input evaluation usually result in the specification of program objec- 
tives, procedure, schedule, and budget by the proposing agency and in 
determinations of whether and at what level to fund the proposal by the 
potential funding agency. 

Process Evaluation 

Once a planned course of action has been chosen and approved 
and the implementation of the plan has begun, process evaluation is * 
needed to provide periodic feedback to project administrators and 
others responsible for continuous control and refinement of plans and 
procedures. The objective of process evaluation is to detect or predict, 
during implementation stages, defects in the design or its implemen- 

4 

tation. The overall strategy is to identify and monitor, on a continuous, 
molar, noninterventionist basis, the potential sources of failure in a 
project. Roger Barker has recently characterized this approach as 
tranducer inquiry in an ecological context. * Here, the evaluator 
does not exercise experimental control over the situation, nor does 

, 4 

he manipulate it in any way. He accepts it as it is and as it evolves, 
and monitors the total situation as best he can by focusing his most 
sensitive noninterventionist data collection techniques on the most 

9 - 4 

| ^Barker, Roger G. "Explorations in Ecological Psychol- 

ogy, 11 American Psychologist, January 1965, 20, 1-14. 
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crucial aspects of the project. Such evaluation is multivariate, and 
not all of the important variables can be specified before a project is 
initiated. The process evaluator focuses his attention on theoretically 
important variables, but he also remains alert to any unanticipated but 
significant events. Under process evaluation, information is collected 
daily, organized systematically, analyzed periodically (e. g. , weekly), 
and reported as often as project personnel require such information 
(e. g. , monthly). Project decision makers are thereby provided with 
information needed for anticipating and overcoming procedural diffi- 
culties. A record of process information is kept for interpreting 
project outcomes, and in longer projects, an updating of context infor- 
mation — providing a dynamic baseline of needs- -is a part of the process 
evaluation. In the change process, process evaluation thus provides 
information for insuring quality control in the immediate and subsequent 
implementation and refinement of plans . 

Product Evaluation 

At the end of the project or at other predetermined times, product 
evaluation is needed for determining the effectiveness of the project. 

The objective of product evaluation is to relate outcomes to objectives 
and to context, input, and process, i.e. , to measure and interpret 
outcomes. The method is to define ope rational criteria and measure 
these as associated with the objectives of the activity; to compare 
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these measurements with predetermined standards, or comparative 
bases; and to make rational analyses and interpretations of the outcomes 
in terms of the recorded context, input, and process information. In 
the change process, product evaluation provides information for de- 
ciding to not change, or continue, ferminate, modify, or refocus a 
change activity, and for linking the activity to other major phases of 
the change process. A product evaluation might show that objectives 
had been satisfactorily achieved and that a developed innovation can be 
continued or is ready to be diffused to schools which need such an in- 
novation (depending, of course, on the decision rules of decision 
makers). In another case, product, context, and process evaluation 
could show that while an objective had not been satisfactorily achieved, 
several components of the program might function differently in another 

instance. 

From this overview of the CIPP model it can be seen that evalu- 
ation activities differ greatly in context, input, process, and project 
assessment phases of the total evaluation process. Both evaluation 
designs and evaluation instruments vary from step to step in their 
applicability and relative effectiveness in providing information. Many 
diverse types of information are needed throughout the decision-making 
process, and a multitude of different institutional and value constraints 
(such as vested interests) limit the ability of evaluation to gather that 
information. In addition, certain of the information may have existed 
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prior to the evaluating effort. Such a model, therefore, demands that 
(1) the model be linked closely to the decision process in the program 
it serves, and (2) the evaluator work intimately with the decision 
maker(s). 

Evaluation and the Decision Process in Title I Programs 
The Title I program is designed to increase the education opportu- 
nities, experiences, and attainment of disadvantaged children through- 
out the nation by adding provisional services to them through local 
educational agencies. National objectives are an integral part of the 
program* s focus as is broad control o ,r er the program's existence 
and support. Yet the Title I effort is specifically controlled and coor- 
dinated at the state level and is implemented at the local level. Thus, 
the decisions for which evaluation information must be supplied will be 
made at local, state, and national levels. 

The loops in Figure 2 illustrate the general decisions and functions 
of evaluation as they may exist in programs which involve local-state- 
national cooperation. A set of feedback control loops delineate the 
national cooperation. A set of feedback control loops delineate the 
flow of evaluation information to levels of decision makers. The loop 
at the right of Figure 2 j'rjws local school activities; the intermediate 
loop, state activities; and the left-hand loop, federal activities. 

Block 1 portrays the local school district's program. This is the 
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iocal context from which needs for. educational change emerge and 
within which the changes to meet these needs must ultimately occur. 

It includes both the intended and actual inputs of the system (e. g. , the 
learners, curriculum, staff, organization, policies, finances, physical 
facilities, and school- community relations), and the intended and actual 
outputs of the system (i. e. , the cognitive, psychological, and social 
functioning of its students and alumni). 

#, 

Proceeding to the right of Block 1, information collection is 
depicted by the first segment of the curved line. This collection 
occurs at the local school level and is essentially objective. Interpre- 
tations of the information are not relevant here, but rather the system- 
atic collection of all information (context, input, process, and product) 
needed for later decisions at local, state, and federal levels. The 
local education agency might independently collect context information, 
and/or it might participate in statewide or national context evaluation, 
e. g. , statewide testing programs, Project Talent, National Assessment. 
In collecting input information, the local education agency probably 
would seek evidence from outside the system about potential solution 
strategies not previously employed within the system; but information 
about potential solution strategies which had previously been tried in 
the system could also, and would probably, be collected locally. 

Process information about a program would be collected daily within the 
system. Product information usually would be collected in the local 
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setting alone at the beginning of a change activity--to provide bench- 
marks --and thereafter on an annual basis* 

Block 2 shows the organization of information. Here information 
would be coded in accordance with a predetermined format, processed 

(e. g. , keypunched), and filed regularly. 

At Block 3, information collected at Block 2 would be analyzed in 
accordance with decision-making requirements for (1) focusing planned 
changes through identifying, explicating, and ranking local system . 
needs (context reports); (2) planning strategies and tactics for effecting 
needed changes (input proposals); (3) directing change activities 
(process reports); and (4) continuing, evolving, and refocusing change 
activities (product reports). Information for context reports to the 
board of education and school administration would be analyzed 
infrequently (perhaps annually or as infrequently as quinquennially) to 
identify find assign priorities to local needs. Input information to 
accompany requests for internal or external support would be analyzed 
in the local system whenever long-range or short-range planning would 
be needed for seeking solutions to local system needs. Once change 
activities were initiated at the local level, process information would 
be analyzed frequently for reports of project staff to determine whether 
potential sources of failure were actually failing, procedures were 
following the project design, projects were on schedule, resources 
were adequate, and the procedural design or its implementation should 
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be modiiied. (This information could also help to diagnose outcomes 
once they were determined. ) Product information would be analyzed 
infrequently for reports to the school administration and the state 
education department to assess the overall effectiveness of the project 
in meeting its general goals and thus to provide information for reas- 
sessing the context, i. e. , how have the problems and needs in the 
local system been modified as a result of the complete change' activity? 

Block 4 denotes program decisions made at the local level. Local 
school decision makers (including the board of education, superintendent, 
principal, supervisor, and/or project director) would use the evaluative 
reports from Block 3 to make decisions required in focusing, planning, 
implementing, and evolving change activities. The board of education 
and school administration would make decisions to focus and set 
priorities for change activities, and they would be aided greatly in this 
by access to context evaluation reports. These persons, together with 
a project director, would use input evaluation information for making 
decisions required in planning, scheduling, staffing, and funding needed 
programs* Following an initial context and input evaluation and the- 

introduction of a change activity, project personnel would use process 

» 

evaluation reports to make decisions required in managing a change 
activity. At infrequent intervals, e. g. , annually, the administration 
would need to decide whether to continue, expand, contract,’ or discon- 
tinue the change activity. While product evaluation reports are 
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important for assisting in such decisions, it should be noted that they 
also are invaluable aids for justifying such decisions to public and staff 
and for honoring those who are responsible for the successes. This 
probably accounts for product evaluation having reached a level of 
development and use which is much higher than that for context, input, 
or process types of evaluation. 

The decisions made at Block 4 would be implemented at Block 5, 
and the school program would undergo frequent modification back at 
Block 1. This cycle is continuous. Evaluation reports would be pre- 
pared as often as decision makers in the local school needed them for 
program examination, planning, control, and evolution. 

Returning to Block 3, product reports for the state education 
department would be prepared annually (or as often as required) by 
all public school districts in the state. At Block 6, the state education 
department would organize these reports into types of projects and 
combine the information from similar projects. This information 
would then be analyzed at Block 7 to determine the strengths and 
weaknesses of the statewide program (state product evaluation), and 
the state program officials would use this information to assess the 
statewide educational problems and needs (context evaluation) to 
decide at Block 8 about program emphases and state control. The 
state's assessment of school product reports to .distinguish between 
effective and ineffective approaches employed by schools would 
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constitute part of the state input evaluation. Decisions by state officials 
would be implemented at Block 9 and would affect the state program 
at Block 10. Local school projects at Block 1 would then be affected 
in turn. 

At Block 7, annual product evaluation from the fifty states would 
be sent to the federal agency. This information would then be organized 
at Block 11 so that major program thrusts could be examined and 
analyzed at Block 12 on a nationwide basis and so that reports (product 
evaluation) could be prepared, for example, for the Associate Com- 
missioner for Elementary and Secondary Education, the Commissioner 
of Education, the Secretary of Health, Education, and Welfare, the 
President, and the Congress. In turn, these functionaries would use 
the product reports to assess the problems and needs of elementary 
and secondary education in increasingly broader contexts (content 
evaluation) and to assess the effectiveness of previous plans and 
programs and the potential of proposed ones (input evaluation). 

Decisions would then be made at the federal leval at Block 1 3 about 
program emphases and funding. The implementation at Block 14 of 
such decisions would affect the federal program at Block 15, the state 
programs at Block 10, and the local school projects at Block 1. 

To summarize the key points related to Figure 2: (1) information 
for evaluation at federal, state, and local levels will be collected 
largely at the local level; (2) this information will form the basis for 
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federal, state and local decisions which will ultimately affect local 
operations; and (3) evaluation plans must be developed, communicated, 
and coordinated at federal, state, and local levels if the information 
schools provide is to be adequate for assisting in the decision process 
at each of these levels. Having presented a model for evaluation and 
illustrated a possible application of the proposed model in a major 
federally- aided program, we will next attempt to depict evaluation as 
conducted by the Evaluation Center within the total context of the 
Columbus Title I program. 

The Columbus Evaluation Program: An Overview 
It is hoped that many useful principles of design for evaluation 
programs may come from collaboration between Columbus and the 
Evaluation Center. The general principles upon which the evaluation 
was based will be outlined shortly; here, a brief overview will be 
presented to provide a frame of reference for viewing these principles. 
When viewing the Columbus evaluation program as a prototype for 
other or further evaluation efforts, the forces which constrain the 
program must be identified and understood. Several limitations on the 
generalizability of the Columbus program appear to exist. These 
constraints, both conceptual and operational, gave shape to the evalu- 
ation and are explained in the following paragraphs. 

The evaluation program began formally, as mentioned earlier in 
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this report, only after the ’’context'* and "input” decisions concerning 
the projects had been made. Thus, the evaluation must remain a test 
of only the "process" and ’’product” components of the CIPP model. 
Nonetheless, the testing of process and product evaluation have had 
relevance for context and input evaluation, for many potential problems 
have been uncovered. 

Staff members for the evaluation effort were provided both by the 
Columbus Public Schools and by the Evaluation Center. The school 
system granted leaves of absence to severcl of its staff members (six 
teachers and a school nurse), in the hope of obtaining qualified 
evaluators to aid in establishing an agency to conduct evaluation 
activities within the system. These people became "project residents, ’’ 
charged with the responsibility of carrying out evaluation activities for 
a particular Columbus project. The Evaluation Center provided per- 
sonnel for the positions of director and associate director of the 
program. Test development specialists, data processing and statistics 
specialists, research assistants, and secretarial staff were funded 
jointly by the two agencies. 

The origins of staff for the program were evident in the backgrounds 
and orientation of staff members. Speaking in the misleading terms of 
stereotypes, the staff represents people with research orientations 
characterized by a quantitative background and approach to problems 
and people with orientations to teaching and experiences with the 



48 



Columbus 
Page 19 



personal concerns of students and teachers. One of the immediate 
changes wrought by the mixing of people of varied interests and training 
at the Evaluation Center was an amalgamation of their different pre- 
dominant views. The result was that each staff member became 
sensitive to the objectives of evaluation and acquired the empathy needed 
to carry it out. 

On the other hand, the implications of staff differences were not 
immediately evident. For example, only through painful experience 
did it become obvious that any future program to train evaluators must 
be designed only after assessing the capabilities and viewpoints of 
trainees and trainers participating in the training. A context and input 
evaluation of the evaluation training program might have pointed out 
the importance of this problem. 

The data generated by the evaluation effort have remained in the 
ownership <of the Columbus Public Schools. Such a policy is in accord 
with the recognition that evaluation ought to be oriented to decision 
makers in the program. (This is explicitly recognized in the concep- 
tual relationship between evaluation and the decision process in the 
Title I program. ) 

General Evaluation Design Principles 
Principles serving as a foundation for the Columbus evaluation 
program are concerned with the design of process and product 
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evaluation activities, the construction and administration of instruments, 
and the manner in which obtained evaluation information is fed back to 
the project staff. 

1. Orientation meetings are held at the beginning of projects and 
periodically during the life of a project . The purpose of these meetings 
is to inform the participants in a project of its objectives, its procedures, 
and its evaluation. At orientation meetings, a special point is made of 
discussing the purpose of evaluation, with emphasis put on the need to 
improve the project by continually evaluating it. By such means, the 
evaluation staff attempts to involve others in the process of evaluation 
and to diminish its threat. Such meetings also give personnel in the 
school system a chance to contribute to the design of a project and its 
evaluation — or at the least to suggest potential barriers to the success 

of a project so that these barriers may be monitored through process 
evaluation. 

2. Initial design of the evaluation effort is a collaborative, activity. 
Objectives for each of the projects in the Title I program are set and 
written by the appropriate project staff from Columbus Public Schools. 
During and after the period of setting objectives, the director ci the 
Title I project under discussion has the responsibility for detailed 
project objectives. The Evaluation Center staff has the responsibility 
to review, comment on, and act on these objectives, striving to clarify 
objectives and couch them in terms of observable behavior. At the 
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same time that objectives for a project are determined, criteria for. 
measuring objective attainment are set for use in product evaluation. 
Thus, an overall design covering each project for a full school year is 
written by the Evaluation Center. Finally instruments to fulfill the 
demar ’ 1 of the design are chosen or constructed by staff members of 
the Center with periodic consultation being provided by the University’s 
Test Development Center. 

3. Process evaluation activities are given focus and defined prior 
to beginning Title I project activities in the schools . The Columbus 
Title I project staff and project residents from the Evaluation Center 
jointly suggested possible barriers’ to the success of the project. These 
barriers then served as the focus of process evaluation, as the project 
was continuously monitored. If' information was generated on other 
barriers to project activities, the process evaluation was then re- 
focused, instruments were revised or new ones constructed, and the 
evaluation proceeded. 

4. Process evaluators are noninterventionists, but process 
evaluation is intervening . A guiding principle in the evaluation was 
that evaluators themselves should not intervene in the operation of a 
project. To meet this objective, much of the needed process informa- 
tion was collected fr*. *\ logs, records which were needed for purposes 
of bookkeeping, and unstructured and conversational interviews. 
However, the assumption upon which process evaluation itself is 
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predicated is that evaluators serve as decision makers, and, to the 
extent that process evaluation causes shifts m project operations, it 
does indeed intervene. Hence, a guiding principle of process evalu- 
ation is that it intervenes to guide the project, although this interven- 
tion is at the discretion of project decision makers. 

Unfortunately, it is especially difficult to determine how, or if, 
process evaluation and evaluators intervene in the project. Evaluators 
who are involved in their jobs, interested in providing process informa* 
tion which reflects the true operation of a project, and concerned that 
data are used to make decisions and verify their appropriateness- - 
such evaluators interject unknown influence into the project's operation. 
Often process evaluation is obtrusive, in spite of efforts to prevent 
intrusion. While such (quickly- designed) devices as multi-project 
interview schedules, the use of non- overlapping samples of respondents, 
and the tight scheduling of process instruments mitigated the impact of 

the evaluation on school personnel^ nonetheless the evaluation did 

, • • 

consume their time and effort as well as that of the evaluator. 

5* One of the guiding principles of this evaluation has been the 
guarante e of anonymity to persons when they respond to instrumen ts. 

An assumption in conducting the evaluation has been that participants in 
any project are competent in their roles. Thus, if problems occur in 

day-to-day operations, or if the hoped-for product is not produced, 

• * , 

some component of the project itself is assumed to be at fault. Tc 
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make this point clear to respondents, it has often been stated (so as to 
diminish the hesitancy of persons to respond) that people are not being 

evaluated. 

The guarantee of anonymity has, in several instances, been a 
point of contention between evaluation and project staffs. The prob- 
ability of contention appears especially great when individuals, or 
characteristics of a project which bear a person's vested interest, are 
identified as barriers to success. At times, project staff have, 
requested information on the identity of persons who responded to 
instruments or provided a particular piece of information. However, 
denial of such requests and reaffirmation of the principle of anonymity 
has proven to be essential to continually obtaining siequate information 
for the Title I program. 

6. Data are obtained by means of sampling . This approach has 
been taken for the sake of insuring some degree of reliability and 
validity and has been employed in both process and product evaluation 
activities. (In addition, sampling served as a strategy to keep percent- 
ages of response to an instrument high, by not overburdening one 
respondent. Often several samples of the same class of respondents 
were used for different instrument administrations. ) Principles of 
random and stratified random sampling were employed when possible 
in the selection of schools and persons to whom instruments would be 
administered. The sampling approach and the rather large number of 
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instruments administered to project participants have made it necessary 
to schedule instruments carefully for administration in the schools. 

In addition, it was found expedient to combine instruments from dif- 
ferent projects into a single jointly-used instrument, when the instru- 
ments attempted to measure similar objectives present in several 
projects. 

7. Both commercially-available and specially- constructed 
instruments are used in th e evaluation. When possible, instruments 
normally administered as a part of the Columbus Public Schools 
testing program are also used. Among the specially- cons true ted 
instruments for the project are depth- study interview schedules for 

• individual children; questionnaires and interview schedules for 

teachers, administrators, and special personnel assigned to projects; 
Q-sort instruments; attitude surveys, rating forms for facilitating 
factors and deterrents to project success; and various logs and record 
forms. Project residents construct or assist in the construction of 
these instruments as part of their training. 

8. Feedback sessions are held monthly during the life of a project 
to inform the project director (and his staff) of the operations of the 
project. Such sessions are attended by project residents from the 
Evaluation Center and by project staff. Process information is reported 
verbally and by other media (films and charts). Project staff members 
are then given an opportunity to ask questions about findings or 
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participate in refocusing the process evaluation. In addition to these 
sessions, meetings are held between Columbus' Assistant Superinten- 
dent for Special Services and the directors of the Evaluation Center 
and the Columbus evaluation program within it. The program (all 
projects), the operation of evaluation activities, and the planning of 
future activities can be discussed at these meetings, while at the same 
time separate channels of communication between the University and 
the school system remain open (through the upper and lower levels of 

the system). 

The flow of evaluation information in the school system's channels 
of communication at times presented a problem. Feedback, while 
designed for all project participants, often reached only the upper 
levels of project staff. Evaluation information was provided to top- 
level decision makers and had to flow from the top of the organization 

downward. When passage of info rmation was impeded, or when only 

certain information was passed as desired, lower-level personnel felt 
they were obtaining little benefit from the time and effort they contrib- 
uted as respondents in the evaluation. It would appear that all evalu- 
ation information ought to be channeled to all project participants in 
some form or other and often enough to retain their sense of involvement 

in the project and commitment to the evaluation. 

Evaluation, after all, can only be timely when access to informa- 
tion is guaranteed to those project staff members who have a "need 
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to know. » It is apparent that the manner in which information is to be 
disseminated must be determined in the early stages of the .valuation, 
for dissemination itself is a strategy for improving activities by 
influencing decision makers. 

The Role of the Evaluator 

The role of the evaluator is a technical one demanding a high level 
of skill in the use of techniques for collecting and organising, analyzing 
and reporting, information concerning the project he is evaluating. 

. Also, the evaluator must be cognizant of the information requirements 
of those persons who must make decisions in and about Title X programs 
and use these requirements in designing and implementing evaluation 

activities. 

However, it appears that the evaluator operating in a school 
system may have a more difficult role to fulfill than is apparent from 

the previous description. The evaluator must provide, for decision- 
making by the project staff, information which is credible, useful, 
valid, timely, and reliable. The evaluator, therefore, must spend 
much of his time working with (or among) project staff. However, it 
is apparent that the project staff is only one group-a part of the local 
education agency— of several groups which must make decisions con- 
cerning Title I activities. It is quite possible that conflicts may arise 
* 

over the nature of information needed. The evaluator may desire to 
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collect information for other levels of decision makers involved in the 
Title I program, e.,g. the State education department, while the project 
staff may dispute tho necessity or the timing of collecting such infor- 

4 

mation. Questions about the credibility and objectivity of evaluators 
almost invariably aril 6 from the evaluator's clients in many quarters 
whenever ths agency being evaluated imposes limitations on the data 

t # 

to be collected. In such a circumstance, resolution of conflict is 
possible only through engendering understanding about what^all relevant 
decision makers need. 

A second potential problem inherent in the evaluator's role is the 
requirement that the evaluator make no decisions himself for the 
project. The evaluator has the ability to subtly shape the path and 
progress of Title I activities by his use of information. Thus, the 
ev alua tor must affirm often after much searching of his conscience 
that he will take a neutral position with respect to the direction of the 
project, except in technical matters of evaluation. To the extent that 

evaluators deviate from this neutrality, unknown intervention in 

• . 

project activities has begun. 

» 

Research Assumptions, Efforts, and Special Studies 
Research into the theory, methodology* usefulness of evalu- 

i 

ation is a key objective of the Evaluation Center's overall strategy for 
evaluation in the Columbus Public Schools. However, while research 
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into the area of evaluation is valued by the Columbus Public School 
system administration, it is certainly true that the administration has 
research interests of its own. Thus, while the schools may value 
research, they must balance all demands for research in the school 
setting, as well as concerning themselves with how true an image of 

the schools ouch research prerents. In short, research in an ongoing 

# * 

evaluation effort appears to be a particularly sensitive endeavor; this 
is especially true when inquiry is not so much concerned with possibil - 
ities (the intent of the experimental investigator, who asks "What would 
happen if. . . ? ") as with actualities (the intent of the aexperimental 
investigator, who asks "What does happen in the real world?"). Most 
of the efforts of the Evaluatioi ''enter- - al though it is not concerned 
with experimental research on the school system*s problems it un- 
covers --have produced context and process data. These data by their 
very nature are threatening, for they portray problem areas, short- 
comings in previo us programs, barriers to progress, and deterrents ■ 
to success. Explicit research efforts of the Evaluation Center 
(discussed shortly) often produced data concerning the way evaluation 
information was utilized arid the effect it had on the programs of 

concern. Such research has the power to threaten and at times may 

« • 
have touched sensitive points in the school system. 

The combination of threat which can come from research efforts, 
and the policy of vesting ownership of data with the Columbus Public 
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Schools, may have caused a general uneasiness about research on 
evaluation in conjunction with the evaluation program. If a general 
principle can be formulated, it is that research efforts must proceed 
tactfully, and proceed only after their rationale and needs are firmly 
established. The alternative to this cautious and rather slow course 
of action is to bargain, before the evaluation, for the type and amount 
of research to be conducted concurrently with evaluation services. 

A number of research projects have been prodvced as an integral 
• part of the Columbus evaluation program. Most of these projects have 
been concerned with the state-of-the-science of evaluation, its potential 
usefulness in the school, and some of the ways in which evaluation may 
be more efficiently conducted. Among those efforts, proceeding or 
completed, are studies of the roles and training of evaluators, the 
use of information management and management information systems 
in education, the relation of evaluation to decision making in planned 
educational change, the effectiveness of item sampling techniques as a 

method of estimating test norms, and a study on the.use and scoring of 

» 

the Color Pyramid Test. A bibliography on evaluation has been com- 
piled with sources ranging from classical research design to contem 

porary views of evaluation. In addition, a critical survey of the 
« * * * ♦ 
literature in evaluation has been completed to serve as a base of 

theory for conceptualising further evaluation efforts. 

Ruth Odgers of the Evaluation Center conducted a brief study 
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on "The Selection, Training, and Functions of ESEA Title I Project 
Evaluators" (available in mimeographed form from the Ohio State 
University Evaluation Center), The objectives of the study were to 
determine: (1) the functions of a project resident or member of an 
educational evaluation staff, (2) the type and amount of experience 
considered essential for performing such functions, (3) the training 
believed to be necessary, and (4) the personal traits believed necessary 
for evaluators. Ah open-ended questionnaire was administered to a 
selected group of key school administrators from the State of Ohio 
and to members of the Evaluation Center staff. 

There appeared to be three general agreed-upon functions of the 
evaluator: (1) planning educational evaluation; (2) implementing 
educational evaluation plans; and (3) disseminating evaluation findings. 
Asa professional educator, the evaluator was expected to assist school 
staff in the process of conceptualization and determination of educational 
goals, plan programs for their attainment, and help school staff assess 
progress at given points in the program. Unique attributes of the 
evaluator's role were (1) the designing of evaluation programs; (2) 
constructing and/or devising instruments for data collection; (3) ana- 
lyzing and organizing data; (4) reporting findings; and (5) giving help 

« • 

in using results for the purpose of implementing, terminating, or 
modifying projects so that goals could be effectively attained. The type 
of experience and training believed to be necessary for carrying out 
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these functions included a Bachelor's degree in education; graduate 
casework in educational research, and preferably a Master's degree; 
and certification in teaching. Personal traits believed important, in 
addition to those customarily- mentioned ones such as a liking for 
people, a quick intellect, stability, cooperativeness, and concern for 
accuracy and objectivity, were flexibility and willingness to compromise 
and take risks; a tolerance for ambiguity; a high frustration level; 
tact; and the capacity to accept and utilize criticism. 

Because of the orientation of the Evaluation Center's efforts to the 
needs of decision makers, it appeared necessary to construct a 
taxonomy of the decisions which project directors would make. Jack 
Ott, now director of the Ohio State University Test Development Center, 
constructed such a taxonomy and tested it by observing the behavior of 
project directors in Columbus during the course of the year. This 
work, serving as the basis for his dissertation, had two major concep- 
tual components. The first component included the premises, anatomy, 
and typology of educational change. The second component included a 
taxonomy of the decision process which was observed to be part of 
planned educational change. From these two conceptual sources, a 
classification of decisions that project directors made was derived 
Three means were used to gather empirical data which would both 
support and interact with the construction of the taxonomy of decisions. 
First, a log was constructed and used by project residents to note the 
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decisions made by project personnel. Second; monthly feedback 

meetings were observed and the decisions made during the course of 
meetings noted. Third, substantive reports of evaluation efforts were 
analyzed to find decision makers* need for information concerning 
cheir premises. To test the usefulness of this decision taxonomy, the 
questions, information needs, and decisions which were made by 
project personnel were summarized by projects. (The outcome of this 
study will soon be available from the Evaluation Center; interim 
findings can be obtained from Jack Ott, Director, Test Development 
Center, The Evaluation Center, The Ohio State University. ) 

More effective means of conducting evaluation activities were 
explored in an experimental study undertaken to determine and compare 
the validity of item sampling with examinee sampling for estimating 
test norms. Examinee sampling was defined as the case in which norm 
statistics were computed from data obtained by administering all items 
in a test to a random sample of examinees; item sampling , on the other 
hand was defined as the case in which one set of norm statistics is 
derived from several non- overlapping sets of test data, each of which 
is based on random samples of items and examinees. 

The study was designed to answer the following questions: 

1. Would item sampling, where each of n examinees is adminis- 
tered only a proportion of the items from a single test, yield 
as good or better a norm estimate as examinee sampling, 
where all items are adminstered to some proportion of the 
examinees? 
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2. Was the effectiveness of item sampling a function of item 
sample size? 

3. Is item sampling as valid for establishing norms for "advan- 
taged" as for "disadvantaged" children? 

4 Within limits, does a large number of examinees taking a small 
number of items each, or a small number of examineesanswering 
many items each, give a more accurate norm estimate (keeping 
the product of examinees x items taken constant)? Two unique 
features of this study are that it employs item samples as small 
as 6 percent of the entire test, and that it involves elementary 

school students. 

The basic study was replicated in two public school populations: 

1665 subjects from 77 fourth-grade classrooms were chosen from 
schools in ’'disadvantaged" neighborhoods, and the same number from 
schools in "advantaged" neighborhoods. The instrument employed was 
the Word Knowledge subtest of the Metropoli tan Elementary Rea_ding 
Tests . Means, standard deviations, and item statistics were computed 

for each item in the item and examinee sample. 

The findings regarding the first study questions indicated the item 
sampling approach performed about as well as, if not better than, the 
examinee sampling approach. These findings support item sampling as 
acceptable, and probably preferable, as an alternative to examinee 
sampling for estimating test norms. Findings regarding the second 
study question are inconclusive: certain findings suggested that the 
accuracy of the estimated means increases as the size of the item 
samples increase; however, other findings suggested that exactly 
the opposite was the case. It also appeared that a combination 
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of small item samples taken by a large sample of examinees produced 
the best norm estimates. These findings will have to be examined 
carefully, however, before conclusions can be drawn. The findings 
are again inconclusive for the third study question; they suggest that 
sampling overall provided a more accurate estimate for the mean and 
variance for the disadvantaged study population than for the advantaged 
one. However, the differential effectiveness of item sample norm 
estimates for advantaged versus disadvantaged study populations was 
slight; thus, it would appear that item sampling as a technique is at 
least as effective as examinee sampling for advantaged and disadvan- 
taged study populations. Regarding the fourth question, it appears 
that a combination of a large number of examinees taking only a few 
items each produces more accurate norm estimates than a combination 
of a small number of examinees taking many items each. (This research 
is more extensively reported in "An Experimental Comparison of Item 
Sampling and Examinee Sampling for Estimating Test Norms, M by 
Thomas R. Owens and Daniel L. Stufflebeam, mimeographed by the 
Ohio State University Evaluation Center. ) 

A feasibility study of the use of information management and 
management information systems was conducted by several members 
of the Evaluation Center staff. The study, much like the one concerned 
with the relation of evaluation to decision making and planned educational 
change, was concerned with the type of information needed for project 
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decision making. In addition, it was t '„ so concerned with information 
needed for decisions at the local school system and state department 
of education levels of administration. Its basic objective was to 
determine the feasibility of anticipating these information needs prior 

to the necessity for making decisions. 

Questions which were asked by decision makers were examined 
both formally and informally. Decision and question content and 
frequency were assessed to determine what, and how often, people felt 
they ’’needed to know. " Two general strategies for providing informa- 
tion emerged from the study: (l) the strategy of anticipating, on a 
stand-by basis, the demands which decision makers would have for 
information; and (2) the strategy of providing a general base of 

information to be used in decision making. 

The major finding of the study thus far, in terms of feasibility, is 
that not enough is known concerning the questions which decision 
makers ask to anticipate the information needs which decision makers 
will have. Evaluators involved in the study now believe they know 
enough to ask questions concerning decisions, but the decisions them- 
selves and their frequency of occurrence are seldom known. It 
appears, and reasonably so, that decision makers do not ask questions 
until they feel a need for information; however, it is also true that 
given the present state of the art of evaluation, a certain amount of 
"lead time" is involved in providing information for decision makers. 
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In addition, it may be part of the evaluator’s role (if he so chooses) to 
attempt to help the decision maker in anticipating the decisions he must 
make in the future. 

It is necessary to balance the requirement that information be 
provided quickly with the requirement that in order to provide informa- 
tion of high certainty a certain amount of time must be consumed in 
gathering, analyzing, and reporting information. Given the necessity 
of trading off between demands for time and accuracy, one strategy 
which appears viable at the present time is to utilize secondary data 
sources rather than primary ones in gathering evaluation information. 
(Some of the thoughts which motivated the conception of this study are 
contained in "Evaluation of Planned Educational Change at the Local 
Education Agency Level, " written by Howard O. Merriman, and 
published as Occasional Paper 67-106 of the Division for the Study of 
Education, School of Education, The Ohio State University. ) 

Two research efforts have been focused on literature which cur- 
rently exists in the field of evaluation. The first work, a bibliography 
on evaluation materials available in the library of the Evaluation Center, 
is concerned with gathering references in education research, organi- 
zation theory, conceptual work in evaluation, innovation, change, 
decision making, methodological works on the design of evaluations, 
the gathering and analyzing of information, and the reporting of 
information to decision makers. This bibliography is constantly 
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updated, and supplements are issued to keep evaluators abreast of 
emerging work in the field. The second work is a paper prepared 
on M A Selective Review of Literature in Evaluation, M by Mary C. Burger 
(mimeographed by the Evaluation Center). The review scans evalu- 
ation literature in order to identify trends, highlights, and emerging 
views of evaluation. The aim of the review is to provide a conceptual 
base upon which Evaluation Center staff members can build further 
theories of evaluation. 

One of the major, but unstructured, thrusts of research in the 
Evaluation Center during the course of this project has been conceptual 
research into the context, inputs, process, and product of evaluation 
as it relates to the field of education. The CIPP model has been 
expanded, explicated, analyzed, and dismantled many times, formally 
and informally. While such work cannot be reported as a particular 
paper or set of guidelines, the conceptualization and re- conceptualization 
occurring at the Evaluation Center is manifested in the efforts and 
papers reported in this discussion. In addition, the CIPP model has 
been expanded in its application, being applied to the Bureau of 
Elementary and Secondary Education's programs in the U. S. Office 
of Education and to the Title III program under ESEA of 1965. 

An area of special interest is the area of instrument development. 
This area has received a considerable amount of the creative and 
administrative energies expended during the past year by the Evaluation 
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Center. In addition to the development of such instruments as ques- 
tionnaires, interview schedules, attitude scales, and reporting and 
accounting forms, several quite unique instruments have been developed 
for use in the Columbus Program. The instruments include a Q-sort 
for use in determining the perceptions of the role and function of 
personnel on various projects from the viewpoint of other personnel; 
an interview and questionnaire schedule for simultaneous questioning 
of a respondent concerning a number of projects with similar compo- 
nents, participants, or objectives; a set of objective scoring techniques 
for the Munster berg Incomplete Stories test, which can be used by 
staff members unskilled in the use of this instrument; a facilitant- 
restraint rating scale, which allows personnel in projects to indicate 
the degree to which certain factors of the project impede or expedite 
the operation of that project; and preliminary conceptual work on 
such instruments as a cultural awareness test and an observation 
scale for use in operating classroom situations. 

In summary, instrument development has been a major thrust of 
the Evaluation Center’s efforts, especially since few instruments are 
available which can gather information required by designs for process 
evaluation. (Additional information and copies of instrumentation used 
in the Columbus project may be obtained from the Evaluation Center. ) 
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EVALUATION IN THE COLUMBUS PUBLIC SCHOOL TITLE I PROGRAM 

The Columbus Title I Projects and Program: An Overview 
A brief look at the context of evaluation in Columbus --the Columbus 
Public Schools' Titled projects and program--may be helpful in as- 
sessing the Evaluation Center's activities. Several characteristics of 
these Title I projects have come to shape the nature of evaluation 
activities. First, while all eight of Columbus' projects focus on 
meeting particular acute needs of the disadvantaged child, they do so 
without full integration of their efforts. The Columbus program is not 
a single project with many component parts but one objective- -a child 
with n qualities. It is rather an array of projects each focused on 
specific problems common to disadvantaged children in certain groups 
(though certain of these programs complement one another). Columbus 
has chosen to work intensively on particular problems of different 
groups, rather than provide what might be a less effective base of 
uniform services for all children in the target population. The impli- 
cation of this choice for evaluation activities is that a "total system" 
view must be taken of individual projects themselves rather than 
viewing all projects together. This also has had an operational impact 
in the Evaluation Center--a separate evaluator (or team) works with 
each project and its director. 

A second characteristic of the project which shaped evaluation 
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activities is that most of the context and input decisions had already 
been made by program designers prior to the involvement of OSUEC. 

The problems were identified, inputs assessed, and the program 
designed before the evaluation effort was launched* However, the 
evaluation staff was involved (often intimately but to varying degrees) 
in setting operational objectives and the criteria for measuring their 
attainment, as well as in formulating process evaluation concepts with 
project staff. 

The General Context of the Projects and Program 
The Columbus Title I effort is made up of eight projects operating 
within the environment of a large metropolitan area. To place the 
program and its evaluation in perspective, Columbus is a city of almost 
600, 000 people in a county of over one million residents. The Columbus 
Public School system reflects this size, as it encompasses approximately 
105,000 students and 112 square miles. (For purposes of comparison, 
Columbus' system is the second largest in the state, after Cleveland, 
and just within the top 20 largest school systems in the country. ) 

i 

The city is plagued with many of the problems which were an 
impetus for passage of the ESEA of 1965. Many of its problems may 
be described by. stereotyped urban- suburban disparities. For example, 
there are a large number of non- whites and low- income groups in the 
central city and a smaller number in the urban fringe and suburbs 
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around the city. However, an atypical aspect of Columbus’ problems 
concerns aspiration. Columbus, the capitol of the state, has an economy 
based largely on government and service industry: a large proportion 
of its labor force is employed by state, county, and local governments, 
by the Ohio State University and other universities, by financial insti- 
tutions, and by other industries which provide services for the region. 
For these reasons (among others), members of minority groups often 
have challenging and satisfying jobs. Many of them are also highly 
educated and are vocal in their concern over the plight of those in need. 
Thus, high levels of aspiration are generated among minority group 
members and "disadvantaged" persons, but frustrated by life in 
Columbus' economy. 

The concentrations of "disadvantaged" families and children in 
the central city and the aspirations of these residents have placed 
pressure on the public school system to provide services and facilities 
to enable these disadvantaged people to obtain the skills they need to 

become participating members of society. 

The Columbus Public School system has structured its organization 
(as well as its Title I projects) in an attempt to meet the educational 
needs of disadvantaged and inner-city children. A Division of Special 
Services has been established and is headed by Dr. Joseph L. Davis, 
Assistant Superintendent for Special Services. It is the purpose of 
this Division to plan for and guide educational activities which are 
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designed for inner-city children. The Assistant Superintendent for 
Special Services, then, has the responsibility for operating and 
evaluating the Title I program in the schools. In addition to this 
organizational feature, the system has recognized the existence of 
several degrees of need for special services and has assigned different 
priorities to these services (this integral part of the Title I program 
will '~3 discussed later). The school system has no division for 
research and development; however, it draws freely on the services 

of Ohio State and other nearby universities for such advice and 

« 

assistance as it may need. 

During the 1965-1966 school year, the Columbus Title I projects 
involved almost 8000 public and over 690 private school students 
clustered in 34 public and 5 private schools across the city. To 
serve the needs of these students, over $2, 150,000 of Federal assistance 
was made available during the 1965-1966 and 1966-1967 school years 
in the form of Title I monies. As a first attempt to create the greatest 
benefit from these monies, the needs of schools were answered. For 
this assessment, categories of need* -Priority I, II, III and IV--were 
established. Each of these categories represented schools in neighbor- 
hoods which exhibit respectively less and less need as determined by the 
measures presented in Title I guidelines. Relevant records on puoil 

locations, achievement, and problems were chosen and used by the 
school administration in this assessment. Following this assessment, 
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overall objectives were set, projects were designed, and resources 
W ere allocated in consideration of the type and degree of need exhibited. 

It is important to note that the evaluation program, as it is discussed 
further in this report, did not come into being until after these decisions 

about need were made. 

Th. Title 1 Projects and Eval uation Instrumentation 
Title I projects operating in the Columbus program include After- 
School Study Centers Projects, Basic Mathematics Improvement and 
Reading Improvement Projects. Elementary Counseling Project, Enrich- 
ment Unit (language arts) Project. Health Service Centers Projects, 
Pre-Kindergarten Project and Regional Service Centers Project. The 
project activities and the staff who perform them are described in the 
following sections. Each of these projects includes activities for 
children and for teachers. While the activities are oriented to the known 
needs of disadvantaged children, their needs are not fuller known 
the projects are evolving. Since many teacher, were not fully aware 
of the breadth and depth of these children's needs, the Columbus pro- 
jects each include an in-service training component designed to increase 
the teacher's understanding of the child and his environment. The in- 
service training program may be, and the effort, of the Evaluation 
Center must be, recognised as source, of feedback to the system and a. 
attempts to tailor education more and more closely to the needs of 
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these children. 

The Pre-Kindergarten Project of the Columbus Public Schools 
is designed to provide an organized program, medical services, and 

social services for socially disadvantaged children who are about to 
enter kindergarten (or first grade the next school year). It is focused 
on children whose home environments suggest great need for pre- 
kindergarten experience. These children are usually four years of age. 
Approximately 85 percent of the children come from homes with in- 
comes of less than $3000 a year. 

The program is designed to enable children to develop an adequate 
set of initial skills and the disposition toward intellectual tasks required 
for educational progress. Strategic objectives include the development 
of perceptual skills, linguistic skills, and a mental set which might be 
called "learning to learn" --including motivating the child to find pleasure 
in learning; developing his ability to give attention to others, to pursue 
purposive action, and to delay gratification of his desires and work for 
more distant goals; and enabling him to view adults as sources of ideas 
as well as sources of approval and reward. In addition, the program 
involves parents by welcoming them to classes for observation and 
having teachers visit with them and discuss the progress of their 
children and the role of the home in this progress. Also, children 
served by this program are given a health examination (including 
physical, dental, and visual components). Immunizations are available 



- 74 - 



Columbus 
Page 4E 

through school health services. Referrals are made to consultants or 
health facilities in the area, and these referrals are followed up by 
social workers, aides, and school nurses. 

The program operates in nearly twenty centers in inner-city 
target areas during the school year (mornings and afternoons). The 
staff assigned to each center consists of a teaching and social service 
team composed of certificated kindergarten- primary teachers, lay 
helpers and volunteers (often residents in the target area), and two 
social workers (usually shared among several centers). Also, health 
service personnel from the Health Center Project (described later) 
provide appropriate services. 

The instruments for the Pre-Kindergarten Project were all devel- 
oped at the Evaluation Center for use in the project. One of these is a 
child study record, used by the teacher to check each child's health, 
muscular coordination, emotional stability, social adjustment, general 
intellectual capacities, art skills, dramatic play, music and rhythms 
and language skills, and a checklist for the treatment of problems which 
the child manifests. (The child study record is so designed that its 
information can be transformed into an optical scanner input sheet and 
be processed by computer. ) Also included are an accounting sheet for 
enrollment and the number of students on waiting lists; a materials 
rating scale for those items supplied for the pre-kindergarten student; 
interview schedules for parents, teachers, and principals involved in 
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the project; an accounting sheet for enrollment and a ttendance records; 
and a supplemental log sheet for successes, problems, and recommen- 
dations which the teacher feels are significant. 

The Enrichment Unit Project is the largest of Columbus' Title 
I projects, serving over forty public and private schools in the inner 
,gtty# There are two major components in the program; a language 
arts program for primary- level pupils; and an administrative structure 
for primary teachers, designed to allow these teachers to make more 
productive use of their time. 

The language arts program is constantly evolving to meet new 
needs of disadvantaged students and to make the most effective use of 
new discoveries concerning these children. Process evaluation infor- 
mation, the observations of participating teachers, study teams 
concerned with reading, and consultants from outside the public school 
system are all involved in this evolution. Specialized materials have 
been introduced which are oriented to the life styles of students, and 
experiences designed to broaden the cultural awareness of disadvantaged 
children have been provided. 

The administrative structure of the program, the "enrichment 
is based on providing an enrichment teacher for every three or 
four primary teachers and oh creating a system of flexible scheduling 
so that enrichment and regular teachers may work together in various 
ways. The enrichment teacher, highly skilled in the teaching of 
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language arts, is used to support the language arts instruction of the 
regular teacher. Other major benefits are freeing the regular teacher 
for more frequent and productive visits to students' homes and allowing 
the regular teacher to grow professionally by having more time '->r 
study and lesson preparations 

Three schedules involving the enrichment and regular teachers 
have been established. One permits the enrichment teacher to assist 
a regular teacher with language arts instruction, spending the same 
proportion of each school day in each classroom. A second schedule 
permits the regular teacher to relinquish her duties to the enrichment 
teacher for one day each week; the enrichment teacher would then 
have responsibility for the entire instructional program in the class- 
room. A third schedule relieves the enrichment teacher of her 

classroom duties. 

Each of these schedules results in different benefits for the 
teachers and students involved. The first allows for general flexibility 
in the classroom by teachers to vary the student-teacher load, to break 
the students into groups for discussion, and to take students on trips 
to engage in outside activities. The second permits the regular teacher 
to engage in home visits.to participate in professional growth activities, 
and to develop lessons and language arts materials needed for her 
students. The third allows the enrichment teacher to assist in the 
development of a language arts program or a program of parent 
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education or to participate in professional growth activities. 

All the instruments, except one, for evaluation of the Enrichment 
Unit Project were developed at the Evaluation Center. The (non- 
standardized) instruments developed include (1 ) a classroom teacher 
interview , to sample the teacher's opinions about the projects's at- 
tainment of its objectives, elicit recommendations for improvement of 
the project, and determine barriers to the success of the project; 

( 2 ) an enrichment unit teacher interview schedule , to obtain the same 
information as above from the enrichment unit teacher; (3) a principal 
interview schedule, eliciting information on communication and co- 
operation in the project, the progress of the home- school program, 
professional growth and language programs (as well as soliciting 
recommendations for the improvement of the project and information 
concerning the barriers to its success); (4) a combined _pr in cipal inter - 
view with pre-mailed questionnaire: (5) a general recommendations 
log for gathering classroom and teacher opinion concerning effective 
and ineffective practices in the enrichment unit; (6) a Q-sort developed 
to examine both the ideal and actual role of the enrichment teacher as 
perceived by enrichment teachers, classroom teachers, and principals; 
(71 several materials rating scales to gather information about the 
organization, distribution, and use of selected materials designed for 
the project; (8) a language program recommendations log for gathering 
information concerning the language program component cf the 
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Enrichment Unit project; (9) a professional growth questionnaire, 
for determining the effect of and reactions of teachers to the professional 
growth meetings; (10) a home-school relations activities inventory a nd 
a home- school relationship recommendations logL_ and (11) a parent 
interview to gather information on parental understanding of the project* 
parental perception of the effectiveness of the project, and parents’ 
opinions concerning home visitations. In addition, the Cooperative 
Primary Tests of Language Achievement (written by Educational 
Testing Service) were used in a pre-test-post-teet design to assess 
pupil achievement in the language program. 

The Regional Services Center Pro ject is designed to improve the 
efficiency and effectiveness of providing educational opportunities for 
disadvantaged students. Service centers have been established for 
regions encompassing five elementary schools where substantial 
numbers of disadvantaged students are enrolled. These regional 
service centers are staffed with subject-matter specialists to provide 
coordinated curricular services in elementary* school science, foreign 
languages, art, music, and physical education. In addition, the centers 
are equipped to provide resource materials for these five curricular 
areas. Activities specific to these curriculum areas are designed with 
the unique needs of the disadvantaged student in mind. Staff members 
are chosen for their quick competence in subject-matter areas and for 
their ability to adapt activities quickly and make them understandable. 
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Approximately 30 schools in the inner city are served by regional 
service centers. Each center is staffed by 20 teachers: five are 
resource teachers and three represent each of the five curricular 
areas. In addition, elementary teachers from the Mathematics Improve- 
ment Project and elementary counselors from the Elementary Counseling 
Project are housed at the centers. The staff members rotate out of 
the centers and through schools in the "region, " usually on a weekly 
basis. 

Each of the evaluation instruments administered to participants 
in the Regional Service Centers Project was constructed by members 
of the Evaluation Center staff. Included among these instruments were 
the following: (1) a classroom perception scale, to collect information 
on the perceptions of teachers concerning the functioning and impact 
of resource teachers (from regional service centers) in the classroom; 
(2) a data collection sheet focusing on the hours spent and number of 
classrooms seen by each resource teacher; (3) questionnaires for 
general resource services, science services, visual arts im provement 
services, foreign language services, music resource services, and 
physical education services , to gather information on the role of the 
resource teacher, as well as information from the resource teacher 
concerning training, coordination of activities, communication, 
services provided, materials used, and recommendations which seem 
significant; and f4) a principal interview focusing on the interaction 
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between the regional services center and the principal's school and the 
utilization and effectiveness of resource teachers provided. 

The Elementary Counseling Project is designed to provide coun- 
seling services for disadvantaged students in the target area of Columbus 
inner city, represented by more than 28 public and private schools. 

N 

The project is also designed to provide counselors who are certified 
and qualified for elementary school work in order to continue and 
expand the project's activities. 

Counseling activities, focused on grades 1 to 6, aim to assist 
each student to better understand himself, see his relations to the 
world around him, and understand his responsibilities in interpersonal 
relations and in school. In addition, counseling serves as a means of 
identifying problems for correction and attempts to improve the 
school's communication with parents. 

Seven counselors comprise the staff of this project. Persons 
with a successful elementary or counseling experience are selected. 
Following an intensive orientation to their jobs, they are assigned to 
rotate through four or five schools. They provide approximately one 
day' s service per week to each school. The counselor's final day each 
week is divided between workirg with other staff and pursuing univer- 
sity coursework toward certification. Similar services are provided 
to private schools, with students referred to the counselor by staff 
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Both standardized and non- standardized instruments were used for 
evaluation activities in the Elementary Counseling Project. The 
standardized instruments employed were the Revised Munsterburg 
Incomplete Stories, a Social Acceptance Scale, and the Color Pyramid 
Test . (In addition, a set of objective scoring guidelines were developed 
for the Munsterberg Stories, so that they could be scored, from tape 
recordings of the administration and subject’s responses, by personnel 
without previous experience in the use of such instruments. ) 

A number of non- standardized instruments were developed for the 
project by the staff of the Evaluation Center. Among the instruments 
developed were (1) accounting sheets for individual counselors (to 
obtain information concerning professional activities which had engaged 
staff members), for project directors (investigating plans for expansion 
and guidelines used for the project), and for parent contacts made by 
counselors for differing purposes; (2) a counselor interview guide , to 
guide interviews (with counselors) which focused on the operation of 
the project and the counselor's perception of it; (3) a facilitant- restraint 
scale to determine factors of the project which made project operation 
easy or impeded it; (4) a Q-sort to determine the ideal and actual roles 
of the counselor as perceived by himself, teachers, and principals 
(together with a sorting sheet to allow the Q-sort to be self-administered 
privately and mailed in for analysis); (5) a principal and teacher ques - 
tionnaire ; (6) a professional growth questionnaire to gather data 
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concerning the information gained from professional growth meetings 
which proved useful for counselors in their daily work; and (7) an_in^ 
ventory of guidance awareness to obtain the perceptions of teachers 

concerning the elementary counseling project. 

The Mathematics Improvement Project is designed to provide a 

structured mathematics program for pupils in the inner city who are 
not achieving at a level commensurate with their ability. Highest 
priority is given to pupils who are farthest below their grade. Both 
elementary and secondary school pupils are included in the target 
population of this project. 

The basic objectives of the program are to increase each child's 
fundamental computational skills, help him develop a vocabulary which 
will aid in understanding and expressing mathematical ideas in his 
daily life, develop his ability to think logically and to solve problems 
in a quantitative situation, and motivate the student to expend greater 
effort to perform consistent with his ability. 

The project includes seven mathematics improvement teachers 
assigned to junior high schools in the inner-city area working with 
grades 7, 8, and 9 and five elementary mathematics teachers assigned 
to regional service centers working with grades 4, 5, and 6. Students 
are identified by their scores on mathematics tests normally used in 
the school system and are subsequently referred to mathematics 
improvement teachers. Schedules are arranged (for example) so that 
junior high teachers w;ork with a different group of underachievers each 
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day of the week, while elementary teachers work in a different school. 
(Time is provided for professional growth activities and planning in 
these schedules). 

The Basic Mathematics Improvement Project, with a number of 
greatly different objectives, required a range of standardized and 
non- standardized instruments for its evaluation. Among the non- 
standardized instruments developed by the Evaluation Center were the 
following: (1) a classroom teacher interview schedule and a mathematics 
improvement teacher interview schedule , to gain insight into two views 
concerning the operation and effectiveness of the project; (2) a material s 
rating scale to determine objectives connected with uses of materials, 
the suitability of materials, the frequency of their use, and suggestions 
for additional materials for the project; (3) attitude scales for teacher 
and pupil assessment of individual students' attitudes toward mathe- 
matics; (4) a mathematics department interview schedule , to determine 
the effectiveness of the project as viewed by teachers of mathematics 
in the schools being served; (5) five forms of a mathematics vocabulary 
and measurement achievement test , to meaure attainment of the product 
objectives of the project; (6) a supplemental data sheet to obtain 
information concerning successes, problems, or recommendations of 
teachers in the areas of math instruction, professional growth, and 
program development; and (7) a student inform a tion form onto which 
all pertinent information concerning the student and his performance 
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on relevant mathematics achievement tests could be entered, set up 
for key punching, and processed by computer. The standardized 
Science Research Associates Mathematics Achievement Test , forms 
A and B, were used to measure pupil achievement in the project. 

The Reading Improvement Project, while similar in design to the 
Mathematics Improvement Project, is broader in scope and involves a 
greater number of students from the inner-city area. The project is 
designed to provide concentrated reading instruction, together with 
individualized assessment and guidance for disadvantaged students 
who are not reading at a level commensurate with their ability. The 
project provides services for over 2500 students from the fourth through 
the twelfth grades, with priority given to students from low- income 
families. 

The project is staffed by reading specialists; one or more of 
these teachers are installed in each inner-city school in the project. 

The reading improvement teachers aid in diagnosing and recording 
each student* s reading limitation and potential. Subsequently, these 
teachers help in grouping students according to the level and nature of 
their reading problems. Individual and group remedial instruction is 
then provided. 

The reading improvement and classroom teachers work jointly to 
provide support for underachievers in reading. In secondary schools 
reading improvement laboratories have been established in conjunction 
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with the project, so that students may work with the reading improvement 
teacher or by themselves to improve their skills. 

Only non- standardized instruments developed by the Evaluation 
Center were used in the evaluation of the Reading Improvement Project. 
Among these instruments were the following: (1) a facilitant- restrain t 
scale to determine factors of project operation which are affecting the 
reading improvement project; (2) a principal interview to gather infor- 
mation about the principal's perceptions of the effectiveness of the 
project; (3) a reading teacher interview guide, including both specific 
multiple- choice and open-ended questions to determine the reading 
teacher's perception of project efficacy; (4) a student interview 
focusing on reading interests; and (5) a teacher questionnaire for 
teachers in the intermediate and secondary grades. 

The After-School Study Centers Project of the Columbus Public 
Schools attempts to creat an atmosphere conducive to studyi in an area 
in which homes seldom provide intellectual and material resources to 
strengthen children's learning experiences. The project is aimed at 
upper elementary, junior high, and senior high students. It is housed 
in some twenty-four centers in schools throughout the inner-city and 
operates two (for elementary students) or three (for junior high and 
senior high students) nights a week, from suppertime until mid-evening. 
Use of the centers, regarded as a privilege rather than a right, hope- 
fully will allow the student to improve his self-image, will stimulate 
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him to value academic work more highly, and will motivate him to learn. 

The centers are staffed by a supervising team composed of two 
competent teachers, cadet principals, or principals and a volunteer 
helping team. Volunteers are secured from a supporting community 
agency (for example, church, service organization, or university) 
and are chosen to provide capable persons who can serve as models 
for disadvantaged students. 

Only non- standardized instrumentation was developed for the 
After School Study Center Project. A student interview was constructed 
to obtain information concerning the reasons for attendance or non- 
attendance at the centers, together with information on the usefulness 
to students of resources available there. In addition, a student 
perception scale was developed to gather data on the students' attitudes 
toward school, toward being assisted with their study habits, and 
toward the services and means of obtaining them which were present 
in the After School Study Centers Project. A te acher super vis_or 
questionnaire was developed to determine what teachers did and thought 
about the operation of the centers. Finally, a similar questionnaire 
was developed for volunteer helpers to determine what activities 
volunteers engaged in during their work at the center and their 
perceptions of the successes and problems of the project. 

The Health Centers Project is designed to expand and refine the 
medical and dental services presently operated by the school system 
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and to bring selected services more closely in contact with disadvantaged 
students. On the assumption that good health is a prerequisite to a 
student's being able to benefit from learning experiences, the Columbus 
Public Schools have provided three health centers to serve the inner- 
city region. These centers provide physical examinations (including 
immunizations and inoculations), dental examinations (including 
follow-up, repair, and restoration), and referral of individual cases 
to medical-dental consultants, as well as providing the medical 
services component of the Pre-Kindergarten Project. 

The Health Centers are staffed by one physician, two dentists 
(serving on a part-time basis), one nurse, and one dental hygienist. 

These persons and their equipment are housed in existing clinics in 
three junior high schools in the target area. Referrals to the centers 
are made by teachers, school nurses, and other medical personnel 
in the area. 

Only non- standardized instruments were administered to the 
personnel of the Health Centers Project; the instruments developed 
included both accounting -type instruments and questionnaires and 
interview schedules. Included in the instruments which were created 
by the Evaluation Center staff were (1) a health center personnel 
interview , concerning staff perceptions of the needs of inner-city 
residents, the facilities of the project, and the emerging barriers to 
project success; (2) a data collection sheet for the project, for recording 
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information concerning the movement of persons through the centers, 
and the type and disposition of their treatment; (3) a parent interview, 
attempting to gather information concerning parents’ perceptions of 
treatment by the health center staff and the degree and kind of com- 
munication between staff and patients; (4) a school nurs e questionnaire, 
focusing on the involvement with and assessment of the project; and 
(5) a teacher awareness questionnaire and (6) principal interview, both 
attempting to gather information on the use of the centers by teachers 
and principals and the impact the operation of the centers had on the 
health problems of students and the operational problems of schools 
and their staff members. 

III. A CRITICAL LOOK AT EVALUATION AND ITS INTENT 

Evaluation to serve decision-making requirements must be valid,. 
reliable , timely, credible, and accessible. To what extent has the 
Columbus evaluation effort achieved these objectives? Unfortunately 
no rigorous and thorough study of evaluation has been made during the 
course of the Columbus project. However, a number of judgments 
can be made about the impact of the process and perhaps about its 
effectiveness in reaching these objectives. Validity and reliability are 
* criteria which all information gathering efforts attempt to meet. The 
objectives which separate the concept of evaluation from inquiry in a 
more general sense, and certainly from "basic" research, are the 
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objectives of timeliness, credibility, and accessibility. It is the latter 
three objectives which are especially important to those attempting 
evaluation in an ongoing school program, for attainment of these 
objectives is necessary in order for evaluation to be useful to decision 

makers operating with time constraints. 

The validity of evaluation efforts is difficult to assess, for only 
at times does an explicit theoretical framework exist on the basis of 
which one can make an evaluation design and appropriate instrumen- 
tation. More often, validity is determined (as it was in Columbus) by 
reference to those variables which decision makers find of interest. 
Because of the selective concerns of various decision makers, only 
some of the many variables which could have been investigated were 
focused upon by evaluators in theColumbus project. When variables 
were focused upon, literature searches were undertaken to provide 
some foundation for instruments developed within the Center. Standard- 
ized tests were assessed for their relevance to the objectives of the 
projects and for their usefulness in providing appropriate information 
quickly. At present, little evidence exists to prove whether these 
tests were relevant, beyond the belief of decision makers. Whether 
they were in fact appropriate to the Columbus Title I effort (valid for 
its disadvantaged students) may not be known until some time in the 
future. The validity of process instruments (the type most often con- 
structed at the Evaluation Center) is even more difficult to ascertain, 
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for the va ria ble s- -thei r content and form-which these instruments 
attempted to measure were constantly evolving throughout the course 
of the project. Process variables investigated were those which 
seemed at the moment to be of importance; their importance in other 
projects or in this project at a later date cannot be known. While it 
might be useful to attempt to relate process variables to, for example, 
organization theory, such determinations must be made after the fact 
for Columbus projects. 

Validity, in terms of the variables which interested decision 
makers, appears to be high. Because of the evolution of the project, 
however, the objectives and processes for attaining them have 
changed, and the only thing which can be said is that the evaluation 
effort appeared to be valid at a particular moment. 

The reliability of the evaluation has not been studied formally. 
Overall, there is no way to determine the likelihood that similar 
results would be obtained from a similar evaluation conducted a second 
time-> except as subjective probability tells us that it is likely. Often, 
when similar instruments were used more than once, results which 
were consistent with earlier results appeared. However, using these 
findings as an indicator of reliability must be done cautiously, for if 
instruments were used more than once, it was likely that they were in 
a different, changed form the second time. Few instruments remained 
unaltered; rather, they evolved as did the project. Many instruments 
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were employed only once, on the other hand, to gather data which 
appeared relevant at one moment, but appeared less useful at later 
phases of the project. In addition, when instruments were used over a 
long period of time, or over wide groups, a setting variable appeared 
to be acting which might have influenced the reliability of findings. 
Because samples of respondents used in the evaluation were such 
tightly-knit groups with all members being in close contact, respon- 
dents to similar instruments tended to give collusive information. 

Group members tended to communicate with one another about the types 
of questions asked by evaluators, and often preconceived notions about 
questions (and preconceived answers) were awaiting the evaluator when 
he came to work with a respondent. Such collusion presents a great 
problem if incorrect assessments prevailed in the minds of respondents. 
The fact that evaluators were unable to determine reliability of findings 
was constantly pointed out to decision makers involved in the evaluation 
effort; the degree to which this precautionary statement was believed, 
or played a role in later decisions, is as yet undetermined. 

Achievement of the objective of credibility can be accomplished 
by two means. First, the decision maker himself may be familiar with 
the area under investigation and can determine (on the basis of his 
technical expertise) the credibility of findings presented by another 
person (i. e. , the evaluator). On the other hand, if the decision maker 
is not intimately familiar with the area under investigation, he often 
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forms notions of credibility based on his assessment of the individuals 
presenting information to him. Both of these means of building credi- 
bility have been used in Columbus. In at least one project decision 
makers were doubtful of the credibility of evaluators and only be- 
grudgingly granted the trustworthiness of information. This situation 
came about through an interaction of familiarity with information on 
the part of decision makers and an uncertainty on the part of both 
decision makers and evaluators as to the exact confines of each role. 

In this specific project, a gradual deterioration of relations between 
the evaluation staff and project decision makers occurred. Presently, 
new methods are being used in an attempt to rebuild credibility in the 
eyes of these decision makers. (However, in such a case it is often 
. necessary that both sides give a bit of ground. ) To rebuild credibility, 
the design and instrumentation for the evaluation effort in the coming 
year is being geared particularly closely to the needs of decision 
makers. At the same time, the precise role of evaluators is being 
defined, so that little contention about who ought to do what can occur. 

By and large, sound working relationships have been achieved 
between evaluators and decision makers of individual projects. It 
would appear that, for credibility to be achieved, the evaluator must 
act in a professional manner, make his commentary on findings and 
analysis precise, and attempt to make his values explicit. It is 
important for the decision maker, conversely, to be precise about his 
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objectives and certain about the type of information he believes useful. 

In this way, the evaluation effort may be less threatening and more 
reliable and valid, and evaluators and decision makers may work 
together closely. 

The requirements for credible and accessible information by all 
decision makers in the project tend to interact to produce possible 
fluctuations in reliability. In the Columbus project, for example, the 
orientation of the evaluation staff was predominantly to top-level 
decision makers in projects. However, evaluators themselves often 
had to obtain information from lower-level decision makers or non- 
participants in the formal project decision-making process. Because 
information obtained from respondents often was not accessible to 
them after it had been analyzed and reported, lower-level decision 
makers and non-participants tended to view the evaluation as not being 
particularly credible, nor its findings accessible for their needs . 

Hence, these persons became disenchanted with the progress of the 
evaluation, and informal feedback sometimes was necessary to insure 
their continuing cooperation. 

Timeliness, one of the most important objectives that evaluation 
must meet, has been only partially met in the evaluation thus far. 
Certainly it must be a common complaint among project administrators 
of all sorts that social scientists tend to be of limited use in obtaining 
information to aid in analyzing pressing problems. From an 
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administrator’s viewpoint, it often appears that by the time social 
scientists have defined the problem, debated methodology, proceeded 
with the collection of data in an acceptable and "clean" manner, analyzed 
the data, and reported their findings the problem which motivated this 
analysis has long since been solved, in a judgmental matter, by a 

decision maker pressed for time. 

The concept of evaluation employed in Columbus is an attempt to 

circumvent this objection. Concerns about absolute validity and 
exceedingly high reliability have been slightly forsaken so as to 
increase the timeliness of and accessibility of information. It is 

# 

important to recognize that any; evaluation effort must come to terms 
with all the objectives of sc and evaluation and achieve a mix reflecting 
the needs of the situation. Seldom can evaluation efforts achieve all 

of the objectives to the fullest degree. 

Timeliness is a major concern in the Columbus evaluation effort. 
The best that can be said thus far, however, is that data provided to 
decision makers were provided periodically; the data were not provided 
continuously, nor always at the exact moment when they were most 
needed. Different modes of returning information to decision makers 
were employed, resulting in different degrees of timeliness. Periodic 
feedback sessions were held rather often for each project; however, 
in most instances an average of two months transpired between the 
time instruments were made ready and administered and the time 
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that a complete report was given to decision makers. In the time 
between, informal means of transmitting information were employed. 

Quite often decision makers would remark when confronted with 
a piece of information "I knew that. " However, the interpretation of 
this is open to debate: it may have been that decision makers "knew” 
in an intuitive fashion; it may have been that they knew on the basis of 
earlier, informal contact with evaluators; or it may have been that 
they simply were defensive or anxious to get on with other things. The 
formal feedback sessions were only somewhat timely; few crucial 
decisions were made on the basis of information first presented there. 

Some sort of determination must be made concerning the rough 
trade-off of timeliness for achievement of other objectives of sound 
evaluation prior to creating the evaluation design for a project. Time- 
liness can be achieved, of course, but at a cost in the attainment of 
other objectives (though it is impossible to say exactly how much of 
one will obviate some of another). The determination of this trade-off 
should not be made by decision makers alone, for often the evaluation 
staff will believe they have a responsibility for the validity and 
reliability of information and will be willing to forego only a limited 
time before becoming adamant that these other objectives be considered. 

Accessibility as an objective of sound evaluation presents an ex- 
ceptionally sensitive issue. A policy about accessibility must be formed 
by decision makers concerning those persons who ’’have a need to know. 
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In Columbus certain respondents to evaluators believed that they ought 
to have some feedback (some accessibility to the information) on the 
results of the evaluation. The determination of how to handle this 
claim cannot be made by evaluators; it must be made by decision makers 
.responsible for the project. 

An even more sensitive problem is that of what might happen 
should unknown persons obtain evaluation information. Public school 
personnel in the Columbus projects were acutely sensitive to the 
desires of interest groups in the city to obtain information about the 
ongoing Title I program. There was often the concern in writing 
repbrts (which the’ Center's philosophy implied ought to go to decision 
makers) that nothing be said which, if it "leaked out, " could damage 
the project. Obviously, determining who has a need to know must be 
done both for persons within and outside the institutional context in 
which evaluation takes place. 

Some mechanics of communication are involved in achieving 
accessibility of evaluation information. Reports need to be written 
in such a way that persons with different backgrounds can read, 
comprehend, and utilize the information contained in them. (At the 
same time, of course, it is important for evaluators to insert technical 
considerations and disclaimers about their data and methods. ) Creation 
of an accessible report and the use of informal communication channels 
were the primary means for achieving accessibility of evaluation 
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information in Columbus. Other means of providing access, such as 
the use of data banks and inventories, were not attempted due to the 
shortage of time, money, knowledge about who had a need to know 
what types of information, and knowledge of how useful such mechanisms 
would be. The present effort appears to be reasonably accessible to 
those persons who, prior to the beginning of the evaluation effort, were 
identified as having a need to know- -the top-level decision makers. 

A concern of evaluation not explicit in the previously-discussed 
criteria is sometimes considered at the Evaluation Center. This 
concern is comprehensiveness--the scope and depth of data which will 
be collected and analyses which will be performed before information 
is presented to decision makers. Decision makers are often quite 
explicit about what they need. They are concerned about the nature 
of a particular problem, the nature of a certain variable, or the inter- 
action of specific elements in the system that they are operating. 

Often, however, it is useful and perhaps significant (in a practical 
political sense) to relate specific problems to long-run or " spillover" 
effects which problems and potential solutions create and to the costs 
of eliminating these effects. Values espoused in the process of rational 
decision making can be narrow or broad on the issue of comprehensive- 
ness. It is important to determine the scope of the evaluation effort 
and the point of view which evaluators shall take in the gathering, 
analysis, and reporting of date, for the simple reason that efficiency 
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is achieved more easily if these values are made explicit. 

As mentioned earlier, the Columbus evaluation is focused on the 
parts of a system; it does not represent a systems analysis of the 
projects, and only a partial analysis of each individual project. The 
evaluation is concerned mainly with information relevant to problems. 

If a total system view appears valuable, the decision maker must 
choose to take it and realize that taking such a view is costly (especially 
since the methodology for gathering information from a comprehensive 
viewpoint is vague). In order to take a comprehensive point of view, 
the scope of comprehensiveness must be defined by decision makers, 
so that the evaluation effort can be precisely focused and problems of 
methodology identified. As this is done, the relation of comprehen- 
siveness to other objectives fora strong evaluation can be more firmly 

defined. 

In summary, the Columbus evaluation effort has been valid to the 
degree that is necessary for decision makers in the project. It has 
been reliable to the degree possible given constraints of time and the 
requirement that instruments constantly change. It has been reasonably 
timely; by and large it has been credible; and it has proved accessible 
to those who were defined as having a need to know. A formal evalua- 
tion of evaluation must be undertaken to provide a description of its 
exact impact on the behavior of decision makers in a project. For the 
moment, it is hoped this informal view and judgmental criticism of 
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evaluation will allow others to begin to make new thrusts in this 
changing field. 
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THE PITTSBURGH EVALUATION PROGRAM 



THE PITTSBURGH EVALUATION MODEL 



Background and Context 

In Pittsburgh Public Schools a model evaluation plan is being 
developed under a contract with the United States Office of Education 
and in conjunction with the evaluation of Title 1 programs. The pro- 
posal for the development of model criteria and procedures in 

Pittsburgh set forth the following premises: 

1. That evaluation in the public school setting is a process for 
program improvement as well as for program assessment 

2. That maximizing the involvement of program personnel 
(field staff and administrative staff) in the process of 
evaluation fosters staff commitment to program improve- 
ment and promotes desired change in staff behavior 

3. That the nondirective role for evaluation personnel is effec- 
tive in promoting program improvement 

In addition, the following conditions or factors in the Pittsburgh 

setting have contributed to shaping the Pittsburgh Model: 

1. Lack of adequate pre- implementation planning and evaluation 

for Title I programs 

Z. Anticipation of eventual support for evaluation of other 

educational programs currently operating within the system 
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3. Proliferation of Title I programs (28 during the first year of 
model development), which has made necessary some com- 
promises between the optimum and the practical in developing 
evaluation procedures 

Out of the foregoing considerations and the experience of model 
building, the following features of the Pittsburgh Model have emerged: 

1. The purpose of program evaluation is seen as that ol pro- 
viding information requisite to program development and 
stabilization and for valid program assessment to those 
responsible for decisions to change the program. 

2. Evaluation and decision-making functions are seen as separate. 

3. Educational programs are viewed as discrete subsystems of 
the total school system. 

4 > The audience for evaluation information is seen as primarily 

the decision makers within each program. The exceptions to 

this rule are at points of inter-program contact (e. g. are 

separate programs compatible? ) and comparison (e. g. which 

1 

is most efficient? ). Decision-making responsibility is 



In order to satisfy the requirements of the Elementary and 
Secondary Education Act of 1965 and of the state and federal agencies 
responsible under the law, product assessment not always consistent 
with the Pittsburgh model is carried on concurrently with model activi- 
ties. Product evaluations not consistent with the model are reported 
annually, whereas evaluation reports under the model are not tied to a 
fixed time schedule. 
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presumed to reside in. all strata of program staff; i«e«t all 
members of program staff, from teachers on up through the 

supervisory and administrative ranks, are potential program 
. developers. 

5. The focus of evaluation under the model- -the evaluation 
criteria and their sequence— reflects an interpretation of the 
information input necessary for program improvement and 
assessment in the public school setting. Program develop- 
ment is seen as a spiraling process, with much recycling of 
change activity as a program evolves. 

6. The procedures of evaluation under the model-- the specifi- 
cations for how evaluation purposes are to be carried out- 
are engineered to encourage the involvement and commit- 
ment of teachers and other categories of field staff to 
insure acceptance and use of evaluation findings. 

The Focus of Evaluation 

The first concern of the model building effort in Pittsburgh is that 
of determining what is to be evaluated. Guidelines have been adopted 
which relate evaluation to the spiral of program development, speci- 
fying both the sequence of evaluation interest over time and the scope 
of interest in point of time. An explanation of the guidelines and their 
underlying rationale is presented in the following section. 
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The Guidelines: Sequence 

Guidelines relevant to sequence in evaluation are represented in 
Figure I . Under these guidelines, evaluation purposes or objectives are 
seen as related to the decision problems of program developers; these 
decision problems, in turn, are seen as being determined by the status 
of the program in the process of program development. For conven- 
ience in formulating a generalized plan of sequence, the development 
process is viewed as being segmented into four levels or stages, and 
a set of evaluation objectives and criteria is associated with each stage. 
In Figure I, the four evaluation stages are listed in Column I, and a 
set of evaluation objectives for each stage in Column II. 

Under the guidelines, each evaluation objective is implemented by 
the evaluation staff through the observation of specified phenomena and 
the application of specified criteria in making evaluation judgments. 

To the extent that it is possible at this time to generalize, the guide- 
lines specify the sources of standards for the criteria. In Figure I, 
the observations and criteria associated with each objective are listed 
in Columns III and IV respectively. The standards sources for speci- 
fied criteria are listed in Column V. The development problems for 
which the evaluation produces information are listed for each stage in 

Column VI. 
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Stage I. The objectives for evaluation in Stage I are to assess 
the viability aud the feasibility of the program design under the condi- 
tion that the program is already operational; the paramount question- - 
which definition, or design, of the program to assess--is resolved by 
the guidelines, which provide for generating a dynamic baseline defi- 
nition, or working definition, of the program by means of public inter- 

2 

views with representative groups of program staff. This observed 
public consensus is then judged for viability by applying the two 
criteria of comprehensiveness and face validity. 

The standard used for the criterion of comprehensiveness has 
been developed in Pittsburgh through systematic analysis of the defi- 
nitions or designs of all Title I programs. This standard is embodied 
in a comprehensive list of program elements and is presented in 
Figure II as the Taxonomy of Program Dimensions. As shown in 

z 

After a Title I program has been implemented in Pittsburgh, 
there are usually at least three designs of the program in existence; 
one is represented by the project proposal, at least one other exists 
in the thinking of program leaders, and a third is reflected by what 
actually happens in the operation of the program. Experience sug- 
gests that, at any given time, the actual number of designs in exist- 
ence for a given program is proportional, inversely, to the quality of 
the intrastaff communication achieved for that program. The rationale 
for the consensus definition is as follows: it provides an expedient 
focus for program development activity aimed at modifying and in- 
ternalizing program goals because (1) it reflects many of the divergent 
views held by the program staff, while at the same time, (2) it is 
largely shaped by the thinking of program leaders whose influence in 
the process of deriving the definition may act as a stimulus for 
inte rnalization. 
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Figure II, program elements are classified into four broad categories. 
Three of these- -outcomes, antecedents, and process- -are regarded 
as necessary and sufficient for program design. The educational pro- 
gram is conceived as a dynamic input -output system; in keeping with 
this concept, channels and procedures for communication are seen as 
essential elements of program design. 

The basic questions relative to comprehensiveness of the pro- 
gram definition are (1) "Is there specific program information for 

each dimension in the taxonomy?" and (2) "Are program dimensions 

» 

spelled out in acceptable form, i. e. , are objectives stated in terms 
which describe behavior, conditions of behavior, and standards or 
criteria?" In judging a design for comprehensiveness in the first or 
second cycles of evaluation, it is common in the Pittsburgh experi- 
ence to find that many of the dimensions are either not specified, not 
complete, or not in desired form (e.‘g. objectives are not stated in be- 
havioral terms). The implication for decision makers in these 
instances is that curriculum development work or other activity to 
extend and/or modify the program definition should be initiated. 

The second criterion for viability of the program definition is that 

3 

A description of the evaluation cycle appears in a later sec- 
tion of this paper. 
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